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GLOSSARY 


Discrimination 

Determination  that  a  detected  object  is  (1)  high  confidence  clutter  (and  need 
not  be  dug),  or  (2)  munitions  or  unknown  (and  must  be  removed) 

Detection 

Determination  of  the  presence  of  a  target,  typically  by  observation  of  a  signal 
level  crossing  a  threshold  set  to  limit  the  probability  that  a  crossing  would  be 
caused  by  noise  or  interference. 

False  Alarm 
(Detection) 

Declaration  of  a  target  that  is  actually  caused  by  noise,  interference,  or 
geology 

False  Alarm 
(Discrimination) 

Declaration  of  an  item  that  could  safely  be  left  in  the  ground  as  munitions  or 
unknown 

Target  of  Interest 

For  this  study,  defined  to  be  intact  munitions,  both  HE  and  practice;  sizeable 
pieces  of  munitions  (on  the  order  of  half  a  round);  and  items  that  look  like 
munitions  (e.g.,  pipes  of  similar  size) 

Characterization 

Determination  of  parameters  that  are  intrinsic  to  a  target  and  can  be  used  to  make  a 
discrimination  decision. 

Classification 

Formally,  determination  that  an  object  belongs  to  a  particular  class  of 
ordnance  (i.e.,  is  a  155  as  opposed  to  an  81).  Classification,  by  its  formal 
definition,  will  not  be  explored  in  this  study.  Instead,  in  this  document  we 
will  use  the  term  classification  as  a  synonym  for  discrimination. 

IX 


DISCRIMINATION  PILOT  PROGRAM 


1.  Introduction 


1.1.  Background 

The  FY06  Defense  Appropriation  contained  funding  for  the  “Development  of  Advanced, 
Sophisticated,  Discrimination  Technologies  for  UXO  Cleanup”  in  the  Environmental  Security 
Technology  Certification  Program.  In  2003,  the  Defense  Science  Board  observed:  “The  . . . 
problem  is  that  instruments  that  can  detect  the  buried  UXOs  also  detect  numerous  scrap  metal 
objects  and  other  artifacts,  which  leads  to  an  enormous  amount  of  expensive  digging.  Typically 
100  holes  may  be  dug  before  a  real  UXO  is  unearthed!  The  Task  Force  assessment  is  that  much 
of  this  wasteful  digging  can  be  eliminated  by  the  use  of  more  advanced  technology  instruments 
that  exploit  modem  digital  processing  and  advanced  multi-mode  sensors  to  achieve  an  improved 
level  of  discrimination  of  scrap  from  UXOs.”  [1] 

Significant  progress  has  been  made  in  discrimination  technology.  To  date,  these  technologies 
have  primarily  been  tested  at  constructed  test  sites,  with  only  limited  application  at  live  sites. 

The  routine  implementation  of  discrimination  technologies  will  require  demonstrations  at  real 
UXO  sites  under  real  world  conditions.  Any  attempt  to  declare  detected  anomalies  to  be 
harmless  and  requiring  no  further  investigation  will  require  demonstration  to  regulators  and 
project  managers  of  not  only  individual  technologies,  but  an  entire  decision  making  process. 

This  discrimination  study  will  be  the  first  phase  in  what  is  expected  to  be  a  continuing  effort  that 
will  span  several  years. 

1.2  Objective  of  the  Demonstration 

1.2.1  ESTCP  Discrimination  Pilot  Program  at  Camp  Sibert 

As  outlined  in  the  Environmental  Security  Technology  Certification  Program  Unexploded 
Ordnance  Discrimination  Study  Demonstration  Plan,  the  objectives  of  the  study  were  to  (i)  test 
and  validate  UXO  detection  and  discrimination  capabilities  of  currently  available  and  emerging 
technologies  on  real  sites  under  operational  conditions,  and  (ii)  investigate  how  UXO 
discrimination  technologies  can  be  implemented  in  cleanup  operations. 

The  broad  study  was  designed  to  test  and  evaluate  the  capabilities  of  various  UXO 
discrimination  processes  which  each  consist  of  a  selected  sensor  hardware,  a  survey  mode,  and  a 
software  based  processing  step.  These  advanced  methods  will  validate  the  pilot  technologies  for 
the  (i)  detection  of  UXO;  (ii)  identification  of  features  that  can  help  distinguish  scrap  and  other 
clutter  from  UXO;  (iii)  reduction  of  false  alarms  (items  that  could  be  safely  left  in  the  ground 
that  are  incorrectly  classified  as  UXO)  while  maintaining  Pas  acceptable  to  all;  and  (iv)  quantify 
the  cost  and  time  impact  of  advanced  methods  on  the  overall  cleanup  process  as  compared  to 
existing  practices. 
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Additionally,  the  study  aimed  to  understand  the  applicability  and  limitations  of  the  selected 
technologies  in  the  context  of  project  objectives,  site  characteristics,  and  suspected  munitions 
contamination.  Finally,  high-quality,  well  documented  data  were  collected  to  support  the  next 
generation  of  signal  processing  research. 

1.2.2  Specific  Objectives  of  the  Demonstration 

The  demonstration  objective  of  our  analysis  was  to  determine  the  discrimination  capability  and 
reliability  of  magnetometer  array  data,  EM61  Mk2  array  data,  GEM  array,  and  EM61  Mk2  cart 
data  using  analysis  procedures  initially  developed  for  MM-0210,  Feature  based  UXO  Detection 
and  Discrimination,  and  MM-0033,  Enhanced  UXO  discrimination  using  Frequency- domain 
Electromagnetic  Induction. 

After  analyzing  each  of  the  four  data  sets  independently,  we  created  six  prioritized  dig  lists. 
Prioritized  dig  lists  were  provided  for  the  EM61  Mk2  Cart  data,  the  GEM  array  data,  and  the 
magnetometer  array  data.  Two  prioritized  dig  lists  were  generated  from  the  EM61  Mk2  array 
data.  The  first  EM61  array  dig  list  was  generated  using  a  solver  that  compares  fit  results  from  a 
constrained,  two-p  target  of  interest  (TOI)  library  with  an  unconstrained  three-p  model.  The 
second  EM61  array  dig  list  was  generated  using  a  solver  embedded  in  UX- Analyze  that  inverts 
using  a  non-constrained,  three-p  model  only.  Finally,  we  combined  inversion  results  from  the 
EM61  Mk2  array  data  and  magnetometer  data  to  produce  a  sixth,  and  final  prioritized  dig  list. 

1.3  Regulatory  Drivers  and  Stakeholder  Issues 

ESTCP  assembled  an  Advisory  Group  to  address  the  regulatory,  programmatic  and  stakeholder 
acceptance  issues  associated  with  the  implementation  of  discrimination  in  the  Munitions 
Response  (MR)  process. 

The  advisory  group  focused  on  exploring  UXO  discrimination  processes  that  will  be  useful  to 
regulators  and  site  managers  in  making  decisions  by  determining: 

•  What  information  is  required  to  support  a  discrimination  decision? 

•  What  does  the  pilot  project  need  to  demonstrate  for  the  community  to  consider  not  digging 
every  anomaly  as  a  viable  alternative? 

•  For  implementation  beyond  the  pilot  project,  how  should  proposals  to  implement 
discrimination  be  evaluated? 
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2.  Technology  Description 


2.1  Technology  Development  and  Application 

We  analyzed  MTADS  magnetometry  data,  MTADS  EM61  MK2  array  data,  MTADS  GEM 
array,  and  standard  cart  EM61  MK2  survey  data.  The  standard  cart  EM61  MK2  survey  data 
were  collected  by  Parsons  Incorporated  using  a  standard  cart  platform.  Typical  industry  standard 
equipment  and  procedures  for  navigation,  geolocation  using  global  positioning  systems  (GPS), 
data  recording,  and  reduction  were  used.  Details  of  the  MTADS  acquisition  systems  and  plans 
are  presented  in  Technology  Demonstration  Plan  entitled  MTADS  Demonstration  at  Camp 
Sibert,  Magnetometer  / EM61  MK2  /  GEM-3  Arrays,  ESCTP  MM-0533.  Because  the  MTADS 
data  acquisition  systems  are  custom  and  not  industry  standard,  a  brief  summary  description 
follows. 

2.1.1  Data  Acquisition  Systems 

The  MTADS  hardware  consists  of  a  low-magnetic-signature  vehicle  that  is  used  to  tow  the 
different  sensor  arrays  over  large  areas  (10  -  25  acres  /  day)  to  detect  buried  UXO  (Figure  2-1). 
Spatial  registration  is  provided  using  high  performance  Real  Time  Kinematic  (RTK)  Global 
Positioning  System  (GPS)  receivers  with  position  accuracies  of  ~5  cm. 

2.1. 1.1.  Magnetometer  Array 

The  MTADS  magnetometer  array  is  a  linear  array  of  eight  Cs-vapor  magnetometer  sensors 
(Geometries,  Inc.,  G-822ROV/A).  The  sensors  were  sampled  at  50  Hz  and  with  a  nominal 
survey  speed  of  6  mph;  resulting,  in  a  sampling  density  of  ~6  cm  along  track.  The  horizontal 
spacing  between  sensors  is  25  cm.  A  single  GPS  antenna  placed  directly  above  the  center  of  the 
sensor  array  measured  the  sensor  positions  in  real-time  (5  Hz).  All  navigation  and  sensor  data 
were  time-stamped  with  Universal  Coordinated  Time  derived  from  the  satellite  clocks  and 
recorded  by  the  data  acquisition  computer  in  the  tow  vehicle. 


Figure  2-1  MTADS  tow  vehicle  and  magnetometer  array 
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2.1. 1.2.  EM61  MK2  Array 


The  EM61  MK2  MTADS  array  is  an  overlapping  array  of  three  pulsed-induction  sensors 
specially  modified  by  Geonics,  Ltd.  based  on  their  EM61  MK2  sensor  with  lxlm  sensor  coils. 
The  sensors  employed  by  MTADS  were  modified  to  make  them  more  compatible  with  vehicular 
speeds  and  to  increase  their  sensitivity  to  small  objects.  The  timing  of  the  gates  has  been  altered 
(Table  2-1).  Differential  mode  was  used  for  this  demonstration.  Nominal  survey  speed  is  3  mph 
and  the  sensor  readings  were  recorded  at  10  Hz.  This  results  in  a  down- track  sampling  of  ~15 
cm  and  a  cross-track  interval  of  50  cm.  In  order  to  obtain  sufficient  “looks”  at  the  anomalies,  or 
to  insure  illumination  of  all  three  principal  axes  of  the  anomaly  with  the  primary  field,  data  is 
collected  in  two  orthogonal  surveys.  The  EM61  array  being  pulled  by  the  MTADS  tow  vehicle 
is  shown  in  Figure  2-2. 

Individual  sensors  in  the  EM61  MK2  array  were  spatially  registered  using  a  three-receiver  RTK 
GPS  system.  An  Inertial  Measurement  Unit  (IMU)  is  also  included  on  the  sensor  array  to 
provide  complementary  platform  orientation  information.  The  IMU  is  a  Crossbow  VG300 
running  at  30  Hz.  A  close-up  view  of  the  sensor  platform  is  shown  in  Figure  2-3  which  shows 
the  three  GPS  antennae  and  the  IMU  (black  box  under  the  aft  port  GPS  antenna). 


Table  2-1  NRL  EM61  MK2  Gate  timing  parameters 


Channel 

4  Gate  1 

Mode 

Differential 

Mode 

Coil 

Delay  (jus) 

Coil 

Delay  (jus) 

i 

Bottom  Coil 

307 

Bottom  Coil 

307 

2 

Bottom  Coil 

508 

Top  Coil 

307 

3 

Bottom  Coil 

738 

Bottom  Coil 

738 

4 

Bottom  Coil 

1000 

Bottom  Coil 

1000 

Figure  2-2  MTADS  EM61  array  pulled  by  the  MTADS  tow  vehicle 
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Figure  2-3  Close-up  of  MTADS  EM61  array  with  GPS  and  IMU 

2.1. 1.3.  GEM-3  Array 

The  MTADS  GEM-3  array  consists  of  three,  96-cm  diameter  Geophex,  Limited  GEM-3  sensors 
in  a  triangular  configuration  with  two  sensors  across  the  front  of  the  array  and  one  centered  in 
the  rear.  The  roughly  2-m  square  array  was  mounted  on  a  rigid  support  which  is  attached  to  the 
MTADS  EM  cart  using  non-metallic  fasteners  (Figure  2-4).  The  GPS  /  IMU  telemetry 
equipment  used  for  GEMTADS  is  the  same  as  that  used  for  the  EM61  MK2  array  and  described 
in  the  previous  section. 

The  standard  GEM-3  sensor  drive  electronics  were  modified  to  produce  a  substantially  higher 
transmit  moment  for  this  array.  Each  individual  sensor  transmits  a  composite  waveform  of  one 
to  ten  frequencies  in  the  frequency  range  of  30  to  20,010  Hz  with  a  base  period  of  1/30  sec.  A 
composite  transmitter  waveform  of  nine  frequencies  log-spaced  from  90  to  20010  Hz  was  used. 
Two  additional  base  periods  were  required  for  signal  deconvolution  and  to  output  the  response 
from  each  sensor.  The  array  operates  continuously  with  one  sensor  actively  transmitting  while 
the  other  two  sensors  are  processing  data  at  any  given  time.  Allowing  for  a  short  coil  settling 
time  between  the  transmissions  from  each  sensor,  an  effective  array  sampling  rate  of  just  over  9 
Hz  was  achieved.  Sequential  transmitter  operation  also  alleviates  the  need  for  the  orthogonal 
survey  mode  employed  for  the  EM61  MK2  array.  Coupled  with  our  standard  survey  speed  of  3 
mph,  the  result  is  a  down-track  sampling  spacing  of  ~15  cm.  The  cross-track  spacing  is  50  cm. 
An  interleaved  survey  pattern  was  used  to  decrease  the  cross-track  spacing  to  25  cm. 


5 


Figure  2-4  MTADS  GEM  array  mounted  on  the  EM  sensor  cart.  In  addition  to  the  three  GEM 
sensors  (false  colors  to  identify  coils),  note  the  three  GPS  antennae  and  the  inertial  measurement 
unit  (under  GPS  receiver  near  back  left  tire)  for  platform  motion  measurement. 

The  GEM  sensors  are  controlled  by  a  custom  electronics  package  designed  and  built  by 
Geophex,  Ltd.  Overall  control  of  data  collection  was  accomplished  with  a  custom  version  of  the 
standard  GEM  sensor  control  software,  WinGem2KArr,  which  logged  the  data  from  the  GEM 
sensors,  the  three  GPS  NMEA  sentences,  the  time  of  the  GPS  1-PPS  pulse,  the  GPS  UTC  time 
stamp,  and  the  IMU  data  in  separate  files  with  a  common  base  survey  name.  The  data  are 
periodically  transferred  to  the  data  analyst  for  immediate  QC  checks  and  for  further  processing. 

2.1.2  Data  Analysis  Methodology 

Our  discrimination  approach  uses  a  model-based  estimation  procedure  to  determine  whether  or 
not  an  unknown  target  is  likely  to  be  a  UXO  item.  It  entails  estimating  the  size  and  shape  of  the 
target  from  the  spatial  pattern  of  the  induced  field  above  the  target  [2,  3].  The  EMI  signal  is  a 
linear  function  of  the  flux  through  the  receiving  coil.  In  our  model,  the  flux  is  assumed  to 
originate  from  an  induced  dipole  moment  at  the  target  location  given  by: 

m  =  UBUtH0 

where  Ho  is  the  peak  primary  field  at  the  target,  U  is  the  transformation  matrix  between  the 
coordinate  directions  and  the  principal  axes  of  the  target,  and  B  is  an  empirically  determined, 
effective  magnetic  polarizability  matrix  [4,  5],  For  an  arbitrary  compact  object,  this  matrix  can 
be  diagonalized  about  three  primary  body  axes  and  written  as: 
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The  relative  magnitudes  of  the  p's  are  determined  by  the  size,  shape  and  composition  of  the 
object  as  well  as  the  transmit  waveform  and  time  gate  or  frequency.  The  transformation  matrix 
contains  the  angular  information  about  the  orientation  of  these  body  axes. 

For  cylindrical  objects  like  most  UXO,  B  is  a  diagonal  matrix  with  only  two  unique  coefficients, 
corresponding  to  the  longitudinal  (Pt)  and  transverse  (Pl)  directions: 


Pl 


0 


Pt 

0 


0 

0 


Pt 


Discrimination  is  based  on  target  P's  estimated  from  spatially  mapped  data.  Specific  ordnance 
items  have  specific  p  values,  while  clutter  items  generally  have  different  P  values. 

We  received  the  final  geo-positioned,  demedianed  magnetometer,  EM61  MK2,  and  GEM-3  data 
from  NRL  and  the  standard  cart  EM61  data  acquired  by  Parsons.  For  each  anomaly  in  each  data 
set,  we  use  E1X- Analyze  to  extract  a  subset  of  data,  taking  care  to  limit  competing  signatures, 
noise  spikes,  and  the  like,  and  invert  using  a  dipole  model.  Fitted  model  parameters  include 
anomaly  size  (based  on  the  moment  for  magnetic  data  and  the  trace  of  the  polarizability  tensor 
for  EMI),  shape  (EMI  only),  XY  position,  depth,  orientation,  and  fit  error  statistics.  In  addition 
to  presenting  the  results  of  the  inversion  in  spreadsheet  form,  UX- Analyze  generated  an  anomaly 
summary  sheet  that  shows  the  measured  data,  inversion  results,  and  model  data  for  QC  purposes. 

2.2  Previous  Testing  of  the  Analysis  Technology 

By  virtue  of  our  involvement  in  a  variety  of  projects,  we  (formerly  AETC  Incorporated)  have 
developed  forward  models  and  inversion  procedures  for  total  field  and  vector  magnetometers  and 
gradiometers,  and  all  of  the  major  EMI  sensors  (EM31,  EM61,  EM61-HH,  EM63,  EM73, 
nanoTEM,  GEM-3).  Furthermore,  as  part  of  several  SERDP  and  ESTCP  projects  we  have 
catalogued  target  parameters  for  an  extensive  inventory  of  UXO  and  clutter  items.  During  the 
Congressionally-mandated  Jefferson  Proving  Ground  Advanced  Technology  Demonstrations, 
AETC  processing  consistently  provided  outstanding  buried  UXO  detection  performance,  and 
only  the  AETC-supported  demonstrations  correctly  classified  more  than  half  of  both  the 
munitions  and  clutter  targets.  Our  analysis  framework  and  procedures,  described  in  a  number  of 
publications  [6-13],  have  been,  or  are  being,  successfully  demonstrated  in  a  variety  of  ESTCP 
Projects,  listed  below. 

199811  Portable  UXO  Detection  System  Adjunct  to  MTADS 

199812  Electromagnetic  Induction  and  Magnetic  Sensor  Fusion  for  Enhanced  UXO 

Target  Classification 

199526  Multi-Sensor  Towed  Array  Detection  System 

199812  Electromagnetic  Induction  &  Magnetic  Sensor  Fusion  for  Enhanced  UXO  Target 

Classification 
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1 999 1 8  Matched  Filter  Processor  for  Detection  and  Discrimination  of  Unexploded 

Ordnance 

20003 1  Airborne  UXO  Surveys  Using  Magnetometer  Arrays 

200033  Enhanced  UXO  Discrimination  Using  Frequency-Domain  Electromagnetic 

Induction 

200034  Demonstration  of  Advanced  UXO  Detection  and  Discrimination  Technologies 

200 108  Handheld  Sensor  for  Unexploded  Ordnance  Discrimination 

2.3  Factors  Affecting  Cost  and  Performance 

Implementation  of  the  demonstrated  analysis  method  requires  additional  time  compared  to  that 
required  for  detection  only.  This  is  due  to  the  fact  that  the  analyst  must  not  only  identify  and 
locate  the  anomaly,  but  also  must  extract  signal  responses  while  excluding  background  or 
overlapping  signatures  to  the  extent  possible,  re-level  the  extracted  data  if  needed,  and  invert 
data  around  the  anomaly  for  model  parameters. 

2.4  Advantages  and  Limitations  of  the  Technology 

This  Demonstration  technology  uses  spatially  referenced  geophysical  data  to  estimate  model 
parameters.  This  has  an  inherent  advantage  over  analysis  methods  based  on  non-quantitative 
schemes.  Non-quantitative  analysis  methods  frequently  base  the  declaration  on  anomaly 
amplitude,  half  width,  spatial  footprint,  or  overall  Took’.  These  characterization  methods  are, 
however,  sensitive  to  the  targets’  orientation  and  depth  of  burial.  The  methodology 
demonstrated  here  separates  the  measured  signatures  into  that  which  is  inherent  to  the  target,  and 
that  which  is  related  to  the  geometry  of  the  problem  (such  as  distance  to  sensor  and  orientation). 

Known  limitations  to  the  data  analysis  approach  adopted  here  result  from  (i)  non-unique 
inversion  results,  and  (ii)  overlapping,  or  non  distinct,  signatures  in  feature  space.  The  former 
limitation,  one  in  which  multiple  sets  of  model  parameters  explain  the  vast  majority  of  the 
observed  data,  is  well  known.  The  second,  while  perhaps  not  as  widely  appreciated,  is  equally 
problematic.  Inverting  EMI  data  using  our  dipole  models,  results  in  three  eigenvalues  of  the 
magnetic  polarizability  tensor,  each  corresponds  to  a  principal  axis  of  object.  Discrimination  is 
possible  only  to  the  degree  that  the  derived  eigenvalues  are  different  for  different  objects  and 
stable  for  similar  classes  of  objects.  In  other  words,  even  with  ideal  data,  the  estimated  burial 
depth,  apparent  size,  and  shape  features  may  not  separate  UXO  and  clutter  signatures  into 
distinct,  non-overlapping  classes.  This  is  because  the  anomaly  features  derived  from  EMI  and 
magnetic  data  are  not  unique  to  UXO.  Clutter  items  that  have  similar  shapes  and  burial 
attributes  to  munitions  can  have  geophysical  signatures  that  are  indistinguishable  from  UXO 
signatures  and,  as  such,  will  have  similar  eigenvalues  and  therefore  likely  be  classified  as 
munitions.  Examples  include  items  such  as  pipes,  post  sections  and  axial  symmetrical 
fragments. 
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3.  Demonstration  Design 


3.1  Performance  Objectives 

Performance  objectives  for  the  demonstration  are  given  in  Table  3-1  to  provide  a  basis  for 
evaluating  the  performance  and  costs  of  the  demonstrated  technology. 

3.2  Selecting  the  Test  Site 

ESTCP  selected  Camp  Sibert  as  the  demonstration  site.  Camp  Sibert  is  located  within  the 
boundaries  of  Site  18  of  the  former  Camp  Sibert.  The  land  is  under  private  ownership  and  is 
used  as  a  hunting  camp. 

The  criterion  that  drove  the  site  selection  process  were  (i)  a  single  use  artillery  or  mortar  range, 
(ii)  simple  clutter  environment,  (iii)  benign  geology,  (iv)  live  munitions  used,  and  (v)  benign 
topography  and  vegetation.  Additional  considerations  were  size  (20-25  acres  was  desired), 
anomaly  density  (mostly  isolated  anomalies;  100-200  per  acre),  total  anomaly  count  (2,500  to 
5,000  anomalies  were  desired),  and  access/authorization  to  seed  site  with  inerted  targets. 

3.3  Test  Site  History/Characteristics 

Information  on  the  former  Camp  Sibert  is  available  in  the  archival  literature  such  as  an  Archives 
Search  Report  (ASR)  developed  in  1993.  The  former  Camp  Sibert  is  located  in  the  Canoe  Creek 
Valley  between  Chandler  Mountain  and  Red  Mountain  to  the  northwest,  and  Dunaway  Mountain 
and  Canoe  Creek  Mountain  to  the  southeast.  Camp  Sibert  is  comprised  of  mainly  sparsely 
inhabited  farmland  and  woodland  and  encompasses  approximately  37,035  acres.  The  City  of 
Gadsden  is  growing  towards  the  former  camp  boundaries  from  the  north.  The  Gadsden 
Municipal  Airport  occupies  the  former  Army  airfield  in  the  northern  portion  of  the  site.  The  site 
is  located  approximately  50  miles  northwest  of  the  Birmingham  Regional  Airport  or  86  miles 
southeast  of  the  Huntsville  International  Airport.  The  site  is  near  exit  181  off  of  Interstate  59  in 
Gadsden  and  located  approximately  8  miles  southwest  of  the  City  of  Gadsden,  near  the  Gadsden 
Municipal  Airport. 
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Table  3-1  Performance  Objectives  for  the  Discrimination  Study 


Type  of 
Performance 
Objective 

Primary  Performance 
Criterion 

Expected  Performance 
(Metric) 

Actual  Performance 
Objective  Met? 

Quantitative 

Correct 

characterization 

parameters 

Radial  Distance:  <25  cm  mean 
and  <  20  cm  standard  deviation 
Depth:  <25  cm  mean  and 
<20  cm  standard  deviation 
Size:  (COV  <0.2) 

Yes  -  excluding  the 
EM61  Cart  data  set 

Maximize  correct 
determinations  of  non¬ 
munitions 

Discrimination  eliminates 
>20%  of  the  detections 
corresponding  to  items  not  of 
interest  at  the  no-dig  threshold 

Yes 

Maximize  correct 
determinations  of 
munitions 

Discrimination  retains  target  of 
interest  above  the  no-dig 
threshold  (>0.95Pd) 

Yes 

Demonstrate  effect  of 
data  quality  (line 
spacing,  location  error, 
signal/noise  levels,  etc) 
on  the  ability  to 
correctly  extract 
parameters  and  classify 
munitions  versus  non¬ 
munitions 

Data  quality  effects  can  be 
isolated  and  understood 

Yes- 

With  the  exception  of 
human  error  (poor 
definition  of  the  spatial 
footprint  of  two  targets), 
the  false  negative  failures 
were  caused  by 
positioning  problems 

*coefficient  of  variation  (COY)  -  stand 

ard  deviation  /  mean 

3.4  Present  Operations 

The  site  is  no  longer  in  active  use  by  the  military.  The  demonstration  area  is  owned  by  a  single 
landowner  who  uses  the  area  for  a  hunting  camp.  The  field  portion  of  the  demonstration  study 
was  conducted  after  January  31st,  the  end  of  the  hunting  season. 

3.5  Pre-Demonstration  Testing  and  Analysis 

No  pre-demonstration  testing  and  analyses  were  performed  against  which  the  performance  of  this 
demonstration  was  compared. 


10 


3.6  Testing  and  Evaluation  Plan 


3.6.1  Scope  of  the  Demonstration 


The  site  is  approximately  25-acres  and  contains  the  following  number  of  anomalies,  by  sensor 


type: 


GEM  Array 

1039 

EM61  MK2  Array 

870 

EM61  MK2  Cart 

663 

Magnetic 

969 

Magnetic  or  EM61  MK2  Array 

1165 


3.7  Management  and  Staffing 

Dr.  Dean  Keiswetter  was  the  project’s  lead  for  SAIC.  Dr.  Tom  Bell  was  the  Quality  Assurance 
Officer.  Dr.  Bruce  Barrow,  Mr.  Jonathan  Miller,  and  Mr.  Tom  Furuya  served  as  Data  Analysts. 


11 


4.  Performance  Assessment 


4.1  Performance  Criteria  and  Confirmation  Methods 

TOIs  were  defined  to  include  (i)  intact  munitions,  both  HE  and  practice,  (ii)  sizeable  pieces  of 
the  munitions  of  interest,  on  the  order  of  half  a  round,  and  (iii)  items  that  look  like  munitions 
(i.e.,  pipes  of  similar  size). 

Items  that  may  be  safely  left  in  the  ground  included  HE  fragments,  single  fins,  cultural  debris, 
geology,  and  small  arms  rounds  (if  present). 

Our  focus  was  on  identifying  items  that  may  be  safely  left  in  the  ground.  The  main  failure, 
therefore,  was  misclassifying  a  TOI  as  an  item  that  can  be  left  in  the  ground.  Individual  targets 
were  scored  as  indicated  in  Table  4-1.  These  discrimination  labels  were  then  used  to  calculate 
the  rolled  up  performance  measures  presented  in  §4.2. 


Table  4-1  Discrimination  Labels  (Categories) 


Category 

Comment 

Recommended  Action: 

(Dig  or  Leave  in  Place) 

1 

High  confidence  clutter 

Leave  in  Place 

2 

Low  confidence  clutter 

Dig 

3 

Can’t  Analyze 

Dig 

4 

Low  confidence  TOI 

Dig 

5 

High  confidence  TOI 

Dig 

The  ‘Can’t  Analyze’  Category  3  label  was  used  to  identify  targets  that  (i)  could  not  be  inverted 
(i.e.,  the  inversion  did  not  converge)  or  (ii)  could  be  inverted  but  the  fit  error  statistics  indicated 
that  the  results  could  not  be  trusted.  We  determined  the  threshold  for  the  later  case  for  each 
dataset  individually  using  labeled  data. 
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4.2  Performance  Criteria  and  Confirmation  Methods 


Our  prioritized  dig  lists  were  scored  against  the  emplaced  targets  by  analysts  from  the  Institute 
for  Defense  Analyses  (IDA).  Performance  results  for  each  of  our  six  prioritized  dig  lists  are 
shown  in  Table  4-2  through  Table  4-7. 

In  the  tables  below,  the  ROC  calculation  excludes  Category  3  items  because  they  were,  by 
default,  recommended  to  be  removed  and  therefore  do  not  contribute  any  gains  due  to  the 
discrimination  process.  Also,  the  metric  used  for  the  estimation  of  size  is  the  coefficient  of 
variation  (COV),  which  is  simply  the  standard  deviation  divided  by  the  mean  value  of  the  size 
parameter.  It  essentially  reports  variability  in  the  estimated  size  parameter.  We  used  this 
normalized  metric  to  compare  performance  because  different  EMI  solvers  return  polarization 
estimates  that  vary  in  absolute  magnitude  and  it  allows  comparisons  of  magnetic  as  well. 


Table  4-2  Performance  and  the  Confirmation  Methods:  EM61  MK2  Cart 


Performance 

Metric 

Expected  Performance 
(pre  demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced) 
at  Pthreshold 

>0.95 

Comparison  to  seeded  items 

0.992 

Pfa  at  Pthreshold 

<0.80 

Validation  Digging 

0.409 

Pfa  at  100%  Pdisc 

<0.80 

Validation  Digging 

0.495 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  Digging 

0.465 

Accuracy  of 
Parameter 
Estimation 

Radial  Distance  (XY):  <25 cm 
Depth:  <15  cm 

SIZE  (UXO  ONLY):  COV  <  0.2 

Comparison  to  truth  data 
Comparison  to  truth  data 
Comparison  to  truth  data 

32cm  (15cm  a) 

15cm  (17cm  a) 

0.18 

a  =  standard  deviation 


Table  4-3  Performance  and  the  Confirmation  Methods:  EM61  MK2  Array  (Approach  1  - 
Library  Based) 


Performance 

Metric 

Expected  Performance 
(pre  demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced) 
at  Pthreshold 

>0.95 

Comparison  to  seeded  items 

0.983 

Pfa  at  Pthreshold 

<0.80 

Validation  Digging 

0.278 

Pfa  at  100%  Pdisc 

<0.80 

Validation  Digging 

0.385 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  Digging 

0.489 

Accuracy  of 
Parameter 
Estimation 

Radial  Distance  (XY):  <25 cm 
Depth:  <15  cm 

SIZE  (UXO  ONLY):  COV  <  0.2 

Comparison  to  truth  data 

22cm  (16cm  a) 

13cm  (14cm  a) 

0.17 
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Table  4-4  Performance  and  the  Confirmation  Methods:  EM61  MK2  Array  (Approach  2  -  Size 
Based) 


Performance 

Metric 

Expected  Performance 
(pre  demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced) 
at  Pthreshold 

>0.95 

Comparison  to  seeded  items 

1.0 

Pfa  at  Pthreshold 

<0.80 

Validation  Digging 

0.421 

Pfa  at  100%  Pdisc 

<0.80 

Validation  Digging 

0.270 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  Digging 

0.470 

Accuracy  of 
Parameter 
Estimation 

Radial  Distance  (XY):  <25 cm 
Depth:  <15  cm 

SIZE  (UXO  ONLY):  COY  <  0.2 

Comparison  to  truth  data 

18cm  (12cm  a) 

20cm  (17cm  a) 

0.28 

Table  4-5  Performance  and  the  Confirmation  Methods:  GEM-3  Array 


Performance 

Metric 

Expected  Performance 
(pre  demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced) 
at  Pthreshold 

>0.95 

Comparison  to  seeded  items 

0.992 

Pfa  at  Pthreshold 

<0.80 

Validation  Digging 

0.389 

Pfa  at  100%  Pdisc 

<0.80 

Validation  Digging 

0.461 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  Digging 

0.482 

Accuracy  of 
Parameter 
Estimation 

Radial  Distance  (XY):  <25 cm 
Depth:  <15  cm 

SIZE  (UXO  ONLY):  COV  <  0.2 

Comparison  to  truth  data 

17cm  (10cm  a) 

15cm  (16cm  a) 

0.10 
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Table  4-6  Performance  and  the  Confirmation  Methods:  Magnetic  Array 


Performance 

Metric 

Expected  Performance 
(pre  demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced) 
at  Pthreshold 

>0.95 

Comparison  to  seeded  items 

1.0 

Pfa  at  Pthreshold 

<0.80 

Validation  Digging 

0.490 

Pfa  at  100%  Pdisc 

<0.80 

Validation  Digging 

0.487 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  Digging 

0.416 

Accuracy  of 
Parameter 
Estimation 

Radial  Distance  (XY):  <25 cm 
Depth:  <15  cm 

SIZE  (UXO  ONLY):  COY  <  0.2 

Comparison  to  truth  data 

14cm  (10cm  a) 

10cm  (12cm  a) 

0.24 

Table  4-7  Performance  and  the  Confirmation  Methods:  Combined  EM61  + 

Magnetic  Array 

Performance 

Metric 

Expected  Performance 
(pre  demo) 

Performance 
Confirmation  Method 

Actual  Performance 
(post  demo) 

Pdisc  (emplaced) 
at  Pthreshold 

>0.95 

Comparison  to  seeded  items 

1.0 

Pfa  at  Pthreshold 

<0.80 

Validation  Digging 

0.305 

Pfa  at  100%  Pdisc 

<0.80 

Validation  Digging 

0.286 

ROC 

0.4 

Gains  over  chance  diagonal 
Validation  Digging 

0.487 

Accuracy  of 
Parameter 
Estimation 

Radial  Distance  (XY):  <25 cm 
Depth:  <15  cm 

SIZE  (UXO  ONLY):  COY  <  0.2 

Comparison  to  truth  data 

NA 

4.3  Data  Analysis,  Interpretation,  and  Analysis 

We  received  sensor  data  and  anomaly  information  (ID  &  location)  from  the  ESTCP  Program 
Office.  Table  4-8  presents  data  sets  and  analysis  details  with  regard  to  training,  model  features, 
and  classifier  for  the  various  data  types.  Each  data  set  is  discussed  individually  in  the  sections 
that  follow. 
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Table  4-8  Detailed  demonstration  approach. 


Training  Data 

Model  Features 

Features  used 
for  Discrimination 

Classifiers 

Gate  or  Frequency 

ro 

TO 

Q 

s_ 

O 

(/> 

C 

d) 

CO 

ro 

3 

T3 

> 

EM61  MK2  CART 
(IDL) 

GPO  &  ESTCP 
directed  digging 

x,y,z,  3  response 
coefficients  for  each  time 
gate,  inclination, 
declination,  roll,  fit  error 

Coherence  Ratio  (library/unconstrained 
normalized  by  fitted  Sdev(SMAX)) 

GLRT 

Both  Coils 

All  three  time  gates  (relative 
weighting) 

EM61  MK2  ARRAY 
(IDL) 

GPO  &  ESTCP 
directed  digging 

x,y,z,  3  response 
coefficients  for  each  time 
gate,  inclination, 
declination,  roll,  fit  error, 
unconstrained,  constrained 

Coherence  Ratio  (library/unconstrained) 

Proximity 

measure 

relative  to  fitted 
line  six  sigma 
below  mean  of 
TOI 

Lower  coil 

All  three  Time  Gates 

GEM  ARRAY 
(IDL) 

GPO  &  ESTCP 
directed  digging 

x,y,z,  3  response 
coefficients  for  each 
frequency,  inclination, 
declination,  roll,  fit  error, 
unconstrained,  constrained 

Coherence  Ratio  (Library/unconstrained 
normalize  by  fitted  Sdev  (SMAX)) 

Proximity 

measure 

relative  to  fitted 
line  three  sigma 
below  mean  of 
TOI 

Quadrature  Only 
(570Hz,  1230Hz,  2610Hz) 

"D 

C 

x,y,z,  3  response 

EM61  MK2  ARRAY 

GPO  &  ESTCP 

coefficients  for  each  time 

Apparent  Size 

PI  DT 

Bottom  Coil 

(UX-ANALYZE) 

directed  digging 

gate,  inclination, 

oLK  1 

Gate  1 

declination,  roll,  fit  error 

MAG  ARRAY 

GPO  &  ESTCP 

x,y,z,  dipole  moment, 

inclination,  declination,  fit 

Magnetic  moment  (size) 

Rank 

NA 

(UX-ANALYZE) 

directed  digging 

error 

T3 

d) 

Lower  Coil 

C 

o 

EM61  MK2  ARRAY 

GPO  &  ESTCP 

see  MTADS  EM61  Mkll 

EM  Coherence  Ratio 

P|  DT 

All  Three  Time  Gates 

E 

PLUS  MAG  ARRAY 

directed  digging 

and  MTADS  Mag  (above) 

Magnetic  Size 

bLK  1 

o 

o 

Mag 
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4.3.1  EM61  MK2  CART 


4.3. 1.1.  Data  Presentation,  Description,  and  Pre-processing 

Data  from  the  EM61  MK2  (Figure  4-1)  is  shown  in  Figure  4-2  thru  Figure  4-4.  The  circles 
identify  anomalies  selected  for  analysis  by  the  ESTCP  Program  Office. 

Summary  of  preprocessing  steps  for  EM61  MK2  Cart  data: 

1)  De-median  filter  applied  to  remove  long  wavelength  drift 

2)  inverted  data  from  both  coils  and  all  three  time  gates  (0.217,  0.366,  and  0.660ms) 

3)  used  tools  developed  in  Oasis  montaj  to  select  polygons  (spatial  footprint)  about 
anomalies 

4)  no  IMU  data  (assigned  roll  and  pitch  to  be  zero) 

One  day’s  worth  of  EM61  cart  data  exhibited  unpredictable  and  unexpected  discontinuities  with 
regard  to  the  time  stamps  and  spatial  positions  (Figure  4-5).  In  the  figure,  this  behavior  is 
evident  by  the  spatial  jumps  in  travel  path  and  gaps  in  time,  which  were  often  on  the  order  of 
1 1  seconds  between  neighboring  data  samples. 

After  preprocessing,  we  ran  inversions  in  batch  and  generated  custom  diagnostic  plots  for  quality 
assurances  purposes.  A  parameter  was  added  to  the  solver  to  automatically  account  for  variable 
time  lag. 


Figure  4-1  Photograph  of  the  EM61  MK2  Cart 
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Figure  4-2  EM61  MK2  Cart  data;  Southwest  area. 
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Figure  4-3  EM61  MK2  Cart  data;  Southeast  area  1. 
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Figure  4-4  EM61  MK2  Cart  data;  Southwest  area  2. 
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Figure  4-5  One  day’s  worth  of  EM61  cart  data  exhibited  unpredictable  and  unexpected 
discontinuities  with  regard  to  the  time  stamps  and  spatial  positions.  This  behavior  is  evident 
from  the  spatial  jumps  in  travel  path  (right)  and  gaps  in  time,  often  on  the  order  of  1 1  seconds 
between  neighboring  data  samples,  as  shown  on  the  left  (highlighted  fields). 


4.3. 1.2.  Inversion  Approach 

Inversion  of  EM61  cart  data  is  based  on  a  standard  dipole  model.  The  algorithm  finds  seven  (7) 
fitted  parameters:  x,y,z,lag,ph,th,ps,  where  x,y,z,  are  the  target  coordinates  (m),  lag  is  an  offset 
between  sensor  and  GPS  clocks  (Figure  4-6),  and  ph,th,ps  are  the  Euler  angles  (deg).  The 
associated  best-fit  betas,  three  polarizations  for  each  frequency,  are  also  recovered  and  values  for 
the  chi-squared  error  and  coherence  are  calculated.  Follow-on  fits  are  made  with  betas 
constrained  to  match  given  values  from  a  library.  Associated  chi-squared  and  coherence  values 
are  found  for  each  library  item,  and  ratios  of  (constrained  coherence)  /  (un-constrained 
coherence)  are  calculated. 

To  avoid  being  trapped  in  local  minima,  the  unconstrained  inversion  operates  in  two  stages.  The 
first  stage  steps  through  fixed  z  values  from  ground  surface  down  to  1.75m  depth  in  5cm 
intervals.  At  each  z  step,  the  best-fit  target  x,y  position  is  found  using  the  a  method  in  which 
elements  of  the  response  tensor  are  found  through  linear  regression.  This  permits  fast  run  times 
and  experience  has  shown  it  is  robust  against  local  minima.  No  restrictions  are  placed  on  the 
tensor  values.  Each  time  channel  was  treated  independently,  so  each  will  have  a  different 
apparent  target  orientation. 
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Target  2020  contour  of  channel  1  with  no  tlm?  offset. 


3456734  5  6  7 

Figure  4-6  The  EM61  MK2  solver  was  modified  to  solve  for  a  constant  time  lag  between  sensor 
and  GPS  clock. 

The  10  best  solutions,  judged  by  chi  square  value,  are  used  as  starting  points  in  the  2nd  stage. 
Before  submitting  these  data,  Euler  angles  must  be  found.  It  was  discovered  that  using  arbitrary 
angles  can  lead  to  failure  of  the  solver  due  to  local  minima,  so  a  special  optimization  step  was 
performed  separately  on  each  of  the  10  solutions.  In  this  step,  “best-fit”  Euler  angles  are  found 
through  minimization  of  off-diagonal  elements  in  the  response  tensor,  across  all  channels. 
Starting  points  for  this  optimization  are  the  apparent  Euler  angles  from  each  channel’s  tensor. 

The  second  stage  searches  all  6  fitted  parameter  values:  x,y,z,ph,th,ps,  through  downhill  simplex 
minimization.  After  cycling  through  all  10  of  the  startpoints  from  the  1st  stage,  the  best-ever 
solution  is  saved. 

Summary  of  solver  steps  used  for  EM61  cart  data: 

•  Standard  Dipole  Magnetic  Polarization  Model 

•  First  stage: 

-  Step  through  Z  (distance  below  array)  in  5cm  increments. 

-  Time  lag  added  as  fit  parameter  to  compensate  for  dynamic  response. 

-  Search  target  X,Y,Lag  nonlinearly,  find  tensor  elements  linearly. 

-  Find  Euler  angles  to  minimize  off-diagonals  across  all  gates. 

-  Save  10  best  fits  judged  by  chi-square  (different  Z  values  &  lags). 

These  are  start  points  for  following  stages. 

•  Second  stage: 

-  Search  x,y,z, phi, theta, psi  all  non-linearly,  find  axial  betas  linearly. 

-  Axial-symmetry  not  assumed. 

•  Third  stage: 

-  Library  of  fixed  beta  values  from  21  UXO  in  GPO  with  high  SNR. 

-  Search  x,y,z, phi, theta  non-linearly  (axial  symmetry  imposed). 

-  Fresh  fit  with  each  library  item  using  10  restarts. 

-  Ratio  of  best  library  fit  coherence  /  unconstrained  fit  coherence  used  in 
discrimination.  This  ratio  is  always  <  1 . 
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4.3. 1.3.  Decision  Metric:  threshold  selection  and  mitigating  factors 

Classification  of  the  EM61  cart  data  was  based  on  results  shown  in  Figure  4-7.  Here,  we  plot  the 
coherence  ratio  (constrained/unconstrained)  versus  fit  mismatch,  or  fit  error.  The  fit  mismatch  is 
defined  as  the  square  root  of  (1 -correlation  coefficient7^).  The  solid  black  line  limits  to  the 
mean  value  of  the  fit  ratio  for  high  SNR  4.2-inch  mortars.  The  dashed  line  is  shifted  down  by 
ten  standard  deviations. 

We  used  a  generalized  likelihood  ratio  test  (GLRT)  to  classify  the  inverted  model  features.  The 
two  features  used  as  input  to  the  classifier  include  the  (i)  coherence  ratio  distance  from  the  solid 
line  representing  the  4.2-inch  mortars,  and  (ii)  a  size  estimate  (the  principal  polarization,  all  three 
time  gates).  The  decision  metric  is  based  the  GLRT  output,  and  set  using  labeled  data  as  follows 
(Figure  4-8): 

Category  1:  <-6.2  (Probability  UXO  <  0.01) 

Category  2:  -6.2  to  1  (Probability  UXO  =  0.01  to  0.5) 

Category  3:  fit  error  greater  than  25% 

Category  4:  1  to  2  (Probability  UXO  =  0.5  to  0.74) 

Category  5:  >2  (Probability  UXO  >  0.74) 

The  threshold  between  categories  2  and  4  set  based  on  first  UXO  encountered.  The  thresholds 
between  1  and  2  and  between  4  and  5  were  set  to  accommodate  observed  uncertainty. 

Mitigating  factors  included  the  following:  (1)  if  the  inversion  failed,  the  anomaly  was  declared 
Category  3,  and  (2)  if  fitted  depth  was  greater  than  0.75m,  it  could  not  be  declared  high 
confidence  clutter  (Category  1). 


Figure  4-7  EM61  MK2  cart  training  data. 


*  UXO  training  set 

o  UXO  GPOEn  library 
x  UXO  GPO  not  library 
a  UXO  Open  field 

*  Cultural 

o  Munitions  Debris 
Half  Round 
Corner  Stake 
Soils 

No  Contact 
o  Rock 


23 


Prioritized  Dig  List 


Figure  4-8  EM61  MK2  cart  training  data  and  selected  thresholds.  Bold  numbers  indicate 
classification  category. 

4.3. 1.4.  Performance  Scores  from  IDA 

Scoring  performances  for  the  EM61  MK2  cart  analysis  are  reported  in  Table  4-9  and  shown 
graphically  in  Figure  4-9.  A  ROC  chart  is  shown  in  Figure  4-10,  where  we  plot  the  number  of 
false  positives  (FP)  versus  the  probability  of  detection  (Pd).  From  a  declared-category 
perspective,  the  Category  3  targets  are  plotted  first,  followed  by  categories  5  (red),  4  (orange),  2 
(yellow),  and  finally  1  (green).  The  colored  dots  on  the  ROC  curves  indicate  the  operating  point 
for  a  Pd=0.95  (red),  the  demonstrator's  threshold  point  (blue)  and  the  lowest  FPdisc  for  Pd=1.0 
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(green),  and  that  the  gray  lines  indicate  a  95%  confidence  interval  around  each  point  on  the  ROC 
curvely. 

Using  the  thresholds  adopted  for  this  analysis,  there  was  one  false  negative.  Anomaly  #1171, 
which  was  in  fact  a  seeded  4.2-inch  mortar,  was  classified  as  high  confidence  clutter  (Category 
1). 

Table  4-9  Performance  Summary:  EM61  MK2  Cart  Constrained/Unconstrained  Analysis 

EM61  CART 


Cultural 

Munition 

Debris 

No 

Contact 

Rock 

Soil 

UXO 

1 

61 

150 

3 

19 

19 

1 

2 

6 

31 

0 

3 

7 

3 

3 

18 

32 

1 

19 

23 

18 

4 

0 

19 

0 

0 

2 

9 

5 

0 

14 

0 

0 

1 

87 

TOTAL  85  246  4  41  52  118 
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Figure  4-9  EM61  MK2  Cart  performance  as  a  function  of  classification  category. 
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<SAIC  EM61  MK2  CART  SUBMITTAL  IDL  SC070925.CSV>,  1244  1209  1231  679,  cart,  Test,  All 


Figure  4-10  EM61  MK2  Cart  ROC  chart;  analyzed  comparing  library-constrained  and 
unconstrained  inversions. 


4.3. 1.5.  Characterization  Plots 

The  mean  error  between  the  fitted  and  measured  XY  locations  for  seeded  4.2-inch  mortars,  based 
on  EM61  MK2  cart  inversion,  is  0.32m  (standard  deviation  of  0. 15m;  Figure  4-1 1). 

The  RMS  depth  error  for  seeded  4.2-inch  mortars,  based  on  EM61  MK2  cart  inversion,  is  0.15m 
(standard  deviation  of  0.17m;  Figure  4-12). 

Figure  4-13  plots  the  net  polarizability,  defined  as  the  sum  of  the  three  principal  polarizations 
(Elog((3))  for  the  first  time  gate  (0.217  ms),  for  each  of  the  119  seeded  4.2in  mortars.  The  seeded 
mortars  were  nearly  identical  in  size  and  shape  and  we  expect  their  net  polarizabilities  to  be 
similar.  For  the  EM61  MK2  cart  data,  the  mean  net  polarizability  was  1.886  with  a  standard 
deviation  of  0.339.  The  COV  for  net  polarizability  is  0.179. 
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Figure  4-11  Differences  between  fitted  and  measured  XY  locations;  EM61  MK2  cart 
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Figure  4-12  Fitted  versus  measured  depth  of  burial;  EM61  MK2  cart 
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Figure  4-13  Raster  of  the  Zp,  which  is  a  measure  of  size,  for  4.2in  mortars,  EM61  MK2  cart 

4.3. 1.6.  Failure  Analyses 

Anomaly  #1171  was  a  seeded  4.2-inch  mortar  but  was  classified  high  confidence  clutter 
(Category  1)  using  the  constrained/unconstrained  approach  detailed  here.  A  reexamination  of 
the  anomaly  data  revealed  that  the  polygon  used  to  define  the  spatial  footprint  of  the  anomaly 
was  inappropriate  and  did  not  include  the  entire  anomaly.  The  decision  metric  for  these  analyses 
was  the  output  of  a  GLRT  classifier  that  used  coherence  ratios  and  principal  polarization 
magnitude  as  inputs.  These  two  inputs  were  chosen  because  they  produced  the  best  performance 
for  the  labeled  data.  Our  post  mortem  analysis  revealed,  however,  that  if  we  had  not  added  the 
principal  polarization  magnitude,  anomaly  #1171  would  not  have  been  assigned  Category  1 
(high  confidence  clutter;  Figure  4-14). 

Anomaly  #506  was  a  seeded  4.2-inch  mortar  but  was  classified  low  confidence  clutter  (Category 
2).  Our  follow-on  analysis  discovered  poor  spatial  coverage  (3  transects  over  anomaly,  0.4m  and 
0.7m  between  adjacent  lanes). 

Anomaly  #8  was  a  seeded  4.2-inch  mortar  but  was  classified  low  confidence  clutter  (Category 
2).  It  too  had  irregular  spatial  sampling  and  conflicting  measured  values.  At  the  intersection  of 
two  transects,  for  example,  the  first  transect  recorded  a  70.4mV  response  while  the  second 
recorded  only  44mV. 

There  were  18  seeded  mortars  assigned  to  Category  3  (i.e.,  Can’t  Analyze).  Of  these,  nine  (9) 
had  fit  errors  above  the  25%  threshold.  The  other  nine  (9)  had  acceptable  fit  errors  but  their 
fitted  depths  were  greater  than  0.75m.  These  later  nine  were  misclassified.  Our  post  mortem 
analysis  uncovered  that  all  targets  with  fitted  depth  greater  than  0.75m  were  assigned  Category  3. 
This  was  a  mistake,  because  our  intent  was  to  simply  not  allow  targets  with  fitted  depths  greater 


29 


coherence  ratio  (libBest/unconstrained) 


than  0.75m  to  be  Category  1  (high  confidence  clutter).  We  intended  to  assign  Category  2  or  4 
for  deeply  buried  targets  depending  on  their  final  discrimination  metric  value  from  the  GLRT 
classifier. 


Fit  mismatch:  sqrt(l-corr(data,model)A2) 


*  UXOtraining  set 
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Figure  4-14  Target  #1171,  identified  by  the  black  diamond,  was  just  above  our  coherence  ratio 
threshold  that  was  established  as  Category  1-2  boundary. 
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4.3.2  EM61MK2  ARRAY 


The  EM61  MK2  Array  data  were  analyzed  independently  using  two  different  inversion 
approaches  and  implemented  in  different  environments.  The  first  approach  utilized  Interactive 
Data  Language  (IDL)  routines  to  perform  unconstrained  and  library-constrained  inversions.  The 
second  approach  utilizes  UX-Analyze  to  perform  unconstrained  3p  model  fits.  Each  approach  is 
described  below. 

4.3.2.I.  Inversions  using  a  Constrained/Unconstrained  Approach 

4.3.2.I.I.  Data  Presentation,  Description,  and  Pre-processing 

Data  from  the  EM61  MK2  array  (Figure  4-15)  is  shown  in  Figure  4-16  thru  Figure  4-18.  The 
circles  identify  anomalies  selected  for  analysis  by  the  ESTCP  Program  Office. 

The  MTADS  MK2  array  platform  consists  of  three  modified  Geonics’s  EM61  MK2  sensors. 

Data  collected  by  the  platform  can  be  modeled  with  the  standard  dipole  response  model,  but 
there  are  several  issues  that  must  be  considered:  anomalous  negative  spikes,  time  varying 
transmit  currents,  inter-calibration  of  the  three  sensors,  and  the  sensor’s  dynamic  response. 

The  negative  spikes  in  the  MTADS  array  data  result  from  an  unfixed  problem  in  Geonic’s 
electronics.  The  electronics  covers  a  large  dynamic  range  by  switching  the  gain  in  the  A-to-D 
conversion;  the  raw  data  is  stored  as  counts  from  the  A-to-D  and  the  current  gain  factor. 
Normally,  as  signals  become  large,  the  gain  factor  switches  automatically.  The  MTADS  system 
has  a  delay  in  this  switching.  As  signals  get  large  (over  most  items  of  interest),  the  A-to-D 
counts,  stored  as  integers,  wrap  from  a  very  large  number  to  a  negative  one.  This  occurs  for  one 
or  two  data  values  and  then  the  gain  switches  and  the  data  proceeds  with  normal  values.  Most  of 
these  spikes  were  fixed  in  pre-processing  by  correcting  any  large  negative  integer  count  values. 
Any  remaining  spikes  were  edited  out  of  the  final  data  set.  This  has  left  small  gaps  in  the  data 
over  some  of  the  anomalies  of  interest. 

The  three  EM61  Mk2  sensors  have  time  varying  transmitter  currents.  As  the  power  supply 
batteries  slowly  go  down  in  voltage,  the  sensors  transmit  smaller  currents.  To  maintain  a 
constant  sensor  output,  each  sensor  measures  the  transmit  current  and  applies  a  gain  based  on  the 
measurement  to  boost  the  outputted  signal.  Hooked  into  an  array,  these  measured  currents 
oscillate  in  time  as  shown  in  Figure  4-19.  Independent  measurements  with  a  calibrated  current 
probe  have  shown  that  these  oscillations  are  not  real.  To  correct  for  this,  the  data  presented  here 
has  been  normalized  by  a  linear  fit  to  the  current. 
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Figure  4-15  MTADS  EM61  MK2  array 
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Figure  4-16  EM61  MK2  Array  data;  Southwest  area. 
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Figure  4-17  EM61  MK2  Array  data;  Southeast  area  1. 
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Figure  4-18  EM61  MK2  Array  data;  Southeast  area  2. 
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Figure  4-19  Measured  EM61  Mk2  array  transmit  currents. 

After  correcting  for  different  transmit  currents,  the  three  array  sensors  still  do  not  have  equal 
responses.  Figure  4-20  plots  the  results  of  moving  a  metal  sphere  over  the  array.  By  symmetry, 
the  left  and  right  sensors  should  have  equal  peaks.  The  red  dashed  lines  indicate  the  expected 
signal  peaks  relative  to  the  center  sensor.  Based  on  this  calibration  data  and  the  relative 
amplitudes  predicted  from  the  model,  a  set  of  relative,  inter-array  calibration  factors  were 
calculated  and  applied  to  the  data. 

Lastly,  Geonics  EM61  sensors  analog  integrate  the  receive  coil  output  and  have  a  dynamic  time 
response  because  of  this.  The  filter  characteristics  of  this  response  can  be  modeled  and  Geonics 
publishes  the  time  constant  parameters  for  this  filter.  The  MTADS  array  has  a  very  short  time 
constant.  To  first  order,  the  filter  simply  shifts  the  data  by  roughly  0.2  seconds.  The  distributed 
data  has  had  this  time  shift  applied.  At  vehicle  speeds  of  1.5  to  2.0  m/s,  the  filter  also  distorts  the 
measured  signal.  We  have  attempted  to  include  this  dynamic  response  in  the  inversion  model, 
but  to  date,  have  not  had  much  success  in  doing  so.  The  inversion  algorithm  has  trouble 
converging  to  the  correct  solution  when  this  filter  is  included.  If  it  does  converge,  the  fit  to  the 
data  is  often  improved.  None  of  the  work  presented  here  accounts  for  the  sensors  time  response. 
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Figure  4-20  Calibration  measurements  of  metal  sphere  over  EM61  MK2  array. 

The  EM61  MK2  array  data  shows  a  great  deal  of  structure  in  the  noise  signal  across  the  site. 
Figure  4-21  shows  contours  of  the  low  signal  levels  in  the  North-South  (NS)  and  East-West 
(EW)  data  sets  across  the  GPO.  The  NS  survey  shows  long  east-to-west  structures  which  are  not 
as  apparent  in  the  EW  survey.  Presumably,  this  noise  is  a  result  of  the  array  crossing  east-to- 
west  furrows  in  the  ground.  It  is  the  result  of  the  array  bouncing  up  and  down  relative  to  the 
ground  surface.  Figure  4-22  and  Figure  4-23  graph  data  along  single  tracks  over  the  GPO  (black 
lines  in  Figure  4-21).  Figure  4-22  shows  the  first  two  channels  (lower/upper  coil,  first  time 
gate).  The  NS  raster  shows  the  up/down  variation  of  crossing  the  presumed  furrows.  The  EW 
raster  shows  a  surprising  amount  of  jittering  noise  on  the  upper  coil.  Presumably,  the  ground 
surface  is  rough  and  the  platform  is  shaking  a  great  deal.  Figure  4-23  plots  the  same  data  for 
channels  three  and  four  (two  later  time  gates  on  the  lower  coil).  Similar  noise  characteristics  are 
apparent  at  smaller  amplitudes.  Figure  4-24  shows  comparison  NS  and  EW  contours  over  a 
section  of  the  “South  West”  site.  The  furrow  structures  are  even  more  apparent.  Figure  4-25 
plots  sample  tracks  for  the  first  two  channels  and  Figure  4-26  plots  the  other  two  channels.  The 
noise  levels  are  significantly  worse  than  the  GPO  across  this  area.  There  are  spikes  on  the  order 
of  +/-  40  mV.  Based  on  these  observations,  it  is  difficult  to  distinguish  actual  detection  signals 
from  structured  noise  with  any  signal  less  than  100  mV. 
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Figure  4-21  EM61  MK2  array  contours  over  GPO  site. 
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Figure  4-22  Rasters  of  channels  #1  and  #2  over  track  samples  on  GPO. 
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Figure  4-23  Rasters  of  channels  #3  and  #4  over  track  samples  on  GPO. 


Contours:  -15,  -10,  -5,  5, 10, 15 

Figure  4-24  EM61  MK2  contours  of  noise  over  South  West  site. 
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North-South  Survey 


Figure  4-25  Rasters  of  channels  #1  and  #2  over  track  samples  on  South  West  site. 
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Figure  4-26  Rasters  of  channels  #3  and  #4  over  track  samples  on  South  West  site. 


40 


Summary  of  preprocessing  steps  for  EM61  MK2  data: 

1)  current  normalization 

2)  inter-sensor  calibration 

3)  inverted  data  from  lower  coils  only,  all  three  time  gates  (0.307,  0.738,  and  1.0ms;  we 
discarded  data  from  upper  coil  due  to  noise) 

4)  used  tools  developed  in  Oasis  montaj  to  select  polygons  (spatial  footprint)  about 
anomalies 

5)  automated  routines  in  IDL  to  carve  out  data  for  each  anomaly,  keeping  data  collected 
along  NS  and  EW  transects  separate 

After  preprocessing,  we  ran  inversions  in  batch  and  generated  custom  diagnostic  plots  for  quality 
assurances  purposes. 

4.3.2. 1.2.  Inversion  Approach 

The  dipole  response  model  for  the  EM61  MK2  array  inverted  the  model  parameters  of:  location 
(Xu,  Yo,  Zo ),  magnetic  polarization  response  terms  (/3X,  fiy,  /3Z),  and  orientation  angles  (yaw,  pitch, 
roll).  The  model  used  here  applied  a  non-linear  Marquardt-Levenberg  search  for  the  location 
parameters  and  did  a  linear  inversion  for  the  general  response  tensor.  The  eigenvalues  of  the 
tensor  are  the  three  response  terms  and  the  eigenvectors  are  the  orientation  angles.  Data  over  a 
given  anomaly  was  carved  out  separately  from  each  survey  (N-S  and  E-W)  and  then  combined 
for  the  fit  process. 

In  an  attempt  to  classify  objects  in  the  <  500  mV  range,  a  second  strategy  was  tried.  The  quality 
of  the  fit  results  was  compared  between  the  previous  unconstrained  3p  fits  with  that  generated 
assuming  fixed  [3  values.  The  second  set  of  fits  was  “library  fits”  with  fixed  response  parameters 
representing  the  UXO  item  of  interest.  The  library  values  were  based  on  the  mean  of  the  log  of 
the  best  unconstrained  fits  (/?s  =  3,  2,  2  for  the  first  time  gate)  from  the  GPO.  The 
discrimination  parameter  becomes  the  ratio  of  the  fit  qualities  between  the  two  fits.  As  this  ratio 
approaches  1.0,  there  is  high  confidence  that  the  source  of  the  anomaly  under  interrogation  is 
munitions. 


4.3.2.I.3.  Decision  Metric:  threshold  selection  and  mitigating  factors 

Classification  of  the  EM61  array  data  was  based  on  results  shown  in  Figure  4-27.  Here,  we  plot 
Fit  Ratio  versus  Fit  Error.  The  Fit  Ratio  is  defined  as  (library  coherence)  /  (unconstrained 
coherence)  and  the  Fit  Error  is  the  square  root  of  (1 -correlation  coefficient^).  The  solid  black 
line  limits  to  the  mean  value  of  the  fit  ratio  for  high  SNR  4.2-inch  mortars.  The  dashed  line  is 
shifted  down  from  the  mean  by  six  standard  deviations  of  the  fit  ratio  for  high  SNR  mortars.  At 
higher  values  of  fit  error,  these  lines  are  arbitrarily  curved  down  to  match  the  higher  fit  error,  low 
SNR  4.2-inch  mortar  data.  Objects  that  are  above  the  dashed  line  are  categorized  UXO  and 
objects  below  the  dashed  line  are  categorized  as  non-UXO,  each  ranked  according  to  their 
distance  from  the  dashed  line.  Distances  above  the  line  are  positive  and  distances  below  the  line 
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are  negative.  The  decision  metric  is  based  on  the  distance  from  dashed  line,  in  units  of 
coherence  ratio,  and  set  using  labeled  data  as  follows  (Figure  4-28): 

Category  1:  <-0.01 

Category  2:  -0.01  to  0 

Category  3:  fit  error  greater  than  0.3 

Category  4:  0  to  0.01 

Category  5:  >0.01 

The  threshold  between  categories  2  and  4  was  set  based  on  first  UXO  encountered.  The 
thresholds  between  categories  1  and  2  and  between  4  and  5  were  set  to  accommodate  observed 
uncertainty. 

Mitigating  factors  included  the  following:  (1)  if  no  appreciable  signal  was  observed  in  either  the 
NS  or  EW  data  set,  the  anomaly  was  declared  a  Category  3,  (2)  if  the  inversion  failed,  the 
anomaly  was  declared  Category  3,  and  (3)  if  an  anomaly  was  present  in  the  NS  transects  only 
(viz.,  no  appreciable  signal  in  the  EW  transects;  Figure  4-29),  it  was  declared  Category  1 . 
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FIT  RATIO 


Diamond-UXO,  X-HalfShell,  Triangle-MD, 
Plus-Cultural,  Asterisk-Rock/Soil 


Figure  4-27  Coherence  fit  ratio  versus  fit  error:  EM61  MK2  Training  +  GPO  data 
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Figure  4-28  EM61  MK2  Array  training  data  and  selected  thresholds. 
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Combined  East-West  Transects  North-South  Transects 


Anomaly 
10m  grid 

Linear  color  scale,  peak  50mV 


Figure  4-29  EM61  MK2  Array  heading  dependent  noise 

4.3.2. 1.4.  Performance  Scores  from  IDA 

Scoring  performances  for  the  constrained  versus  unconstrained  analysis  approach  are  reported  in 
Table  4-10  and  shown  graphically  in  Figure  4-30.  A  ROC  chart  is  shown  in  Figure  4-31,  where 
we  plot  the  number  of  false  positives  (FP)  versus  the  probability  of  detection  (Pd). 

Anomaly  #420,  which  was  in  fact  a  seeded  4.2-inch  mortar,  was  classified  as  high  confidence 
clutter  (Category  1). 

Table  4-10  Performance  Summary:  EM61  MK2  Constrained/Unconstrained  Analysis 

EM61  Array  IDL _ 


CAT 

Cultural 

Munition 

Debris 

No 

Contact 

Rock 

Soil 

uxo 

1 

79 

211 

4 

26 

124 

1 

2 

0 

12 

0 

0 

3 

1 

3 

27 

35 

0 

19 

34 

0 

4 

0 

23 

1 

0 

4 

8 

5 

0 

11 

0 

0 

2 

109 

TOTAL  106  292  5  45  167  119 
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EM61  MK2  Array  Scoring  Performance  (IDL) 
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Figure  4-30  EM61  MK2  Array  performance  as  a  function  of  classification  category. 
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<SAICEM61  MK2  ARRAY  SUBMITTAL  IDL  SC070925.CSV>,  751  320  1170  507,  em61,  Test.  All 


Figure  4-31  EM61  MK2  Array  ROC  chart;  analyzed  comparing  library-constrained  and 
unconstrained  inversions. 


4.3.2.I.5.  Characterization  Plots 

The  mean  error  between  the  fitted  and  measured  XY  locations  for  seeded  4.2-inch  mortars,  based 
on  EM61  MK2  array  inversion  results,  is  0.22m  with  a  standard  deviation  of  0.16m.  Figure  4-32 
plots  the  distance  between  inverted  and  measured  locations  for  UXO  and  clutter,  segmented  by 
discrimination  ranking. 

The  RMS  depth  error  for  seeded  4.2-inch  mortars  is  0.13m  with  a  standard  deviation  of  0.14m. 
Figure  4-33  plots  the  fitted  depth  versus  measured  depth  for  UXO  and  clutter,  segmented  by 
discrimination  ranking. 

Figure  4-34  plots  the  net  polarizability  for  the  first  time  gate  (0.217  ms)  bottom  coil,  for  each  of 
the  119  seeded  4.2in  mortars.  The  seeded  mortars  were  nearly  identical  in  size  and  shape  and  we 
expect  their  net  polarizabilities  to  be  similar.  For  the  EM61  MK2  array  data  and  solver 
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Fitted  Y  -  Measured  Y  (m)  Fitted  Y  -  Measured  Y  (m)  Fitted  Y  -  Measured  Y  (m) 


employed  for  this  analysis,  the  mean  net  polarizability  was  0.682  with  a  standard  deviation  of 
0.1 18.  The  corresponding  COY  is  0.174. 
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Figure  4-32  Differences  between  fitted  and  measured  XY  locations;  EM61  MK2  array 
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Figure  4-33  Fitted  versus  measured  depth  of  burial;  EM61  MK2  array 
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Figure  4-34  Raster  of  the  Zp,  which  is  a  measure  of  size,  for  4.2in  mortars,  EM61  MK2  array 

4.3.2.I.6.  Failure  Analyses 

Anomaly  #420  was  a  seeded  4.2-inch  mortar  but  was  classified  high  confidence  clutter  (Category 
1)  using  the  constrained/unconstrained  approach.  A  reexamination  of  the  anomaly  data  revealed 
that  one  sensor  transect  had  inconsistent  elevation  measurements  relative  to  nearby  transects 
(Figure  4-35).  The  poor  z  positioning  of  this  single  track  of  data  altered  the  inverted 
polarizations  such  that  it  was  not  fit  well  using  our  library  representing  4.2-inch  mortars. 


Figure  4-35  Contours  of  measured  elevation  (z)  show  poor  consistency  for  one  transect  near 
anomaly  #420. 


50 


Anomaly  #192  was  also  a  seeded  mortar  but  classified  as  low  confidence  UXO  (Category  2). 
Upon  inspection,  we  found  that  our  search  algorithm  did  not  search  deep  enough  in  the  depth 
parameter.  Extending  the  range  of  our  search  algorithm  changed  the  classification  of  this 
anomaly  to  high  confidence  UXO  (Category  5). 

4.3.2.I.7.  Summary  and  Comments 

Analyzing  the  EM61  MK2  array  data  using  a  library-constrained  versus  unconstrained  coherence 
metric  resulted  in  one  false  negative.  The  analysis  failure  was  caused  by  poor  elevation  (z) 
positioning  along  one  of  transects. 

Data  glitches  and  timing  problems  degraded  some  of  our  inversions.  A  number  of  anomalies 
acquired  during  a  single  sortie,  for  example,  possessed  timing  errors  and  chevron  signals.  Of  the 
19  fits  from  this  sortie,  six  high  amplitude  anomalies  were  classified  Category  4.  In  another 
case,  #421)  there  are  three  samples  at  identical  spatial  positions,  but  the  EM  signal  is  changing. 
Still  others  (#331  and  #342)  possess  timing  irregularities. 
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4.3.2.2.  Inversions  using  an  Unconstrained  Approach  only  (UX-Analyze) 

4.3.2.2.1.  Data  Presentation,  Description,  and  Pre-processing 

The  EM61  MK2  Array  data  were  presented  and  discussed  in  §4.3.2. 1  above. 

The  preprocessing  steps  for  EM61  MK2  data,  prior  to  analysis  using  UX-Analyze,  included: 

1)  current  normalization 

2)  inter-sensor  calibration 

3)  inverted  data  from  lower  coil  only,  first  time  gates  (0.307  msec) 

4)  used  tools  developed  in  Oasis  montaj  to  select  polygons  (spatial  footprint)  about 
anomalies 

5)  automated  UX-Analyze  tools  to  carve  out  data  for  each  anomaly,  keeping  data  collected 
along  NS  and  EW  transects  separate. 

After  preprocessing,  we  ran  inversions  in  batch  and  generated  custom  diagnostic  plots  for  quality 
assurances  purposes. 

4.3.2.2.2.  Inversion  Approach 

The  dipole  response  model  for  this  analysis  method  inverts  the  model  parameters  of:  location 
(X0,  Yo,  Zo),  magnetic  polarization  response  terms  (J3X,  [5V,  (3Z),  and  orientation  angles  (yaw ,  pitch, 
roll).  The  model  used  here  applied  a  non-linear  Marquardt-Levenberg  search  for  all  nine 
parameters.  For  the  Southeast  1  and  2  areas,  data  from  NS  and  EW  transects  were  merged  prior 
to  fitting.  For  the  Southwest  area,  inversions  were  preformed  on  data  collected  along  EW 
transects  only  due  to  noise  issues  with  the  NS  data.  The  goodness  of  fit  metric,  e ,  is  the  squared 

correlation  of  data  and  model.  By  definition,  the  dipole  fit  error  metric  is  equal  to  Vl  -s  ■ 

4.3.2.2.3.  Decision  Metric:  threshold  selection  and  mitigating  factors 

Using  UX-Analyze,  classification  of  the  EM61  array  data  was  based  on  the  sum  of  the 
polarizations  (E(3).  Figure  4-36  presents  fitted  results  for  all  labeled  data.  In  the  figure,  the  data 
are  prioritized  according  to  apparent  size  and  color  coded  according  to  the  actual  labels.  Based 
upon  these  labeled  data,  we  established  the  following  thresholds: 

Category  1 :  <0.04 

Category  2:  0.04  to  0.05 

Category  3:  fit  error  greater  than  45% 

Category  4:  0.05  to  0.07 
Category  5:  >0.07 

The  threshold  between  categories  2  and  4  was  set  based  on  first  UXO  encountered.  The 
thresholds  between  categories  1  and  2  and  between  4  and  5  were  set  to  accommodate  observed 
uncertainty. 
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Mitigating  factors  included  the  following:  (1)  if  no  appreciable  signal  was  observed  in  either  the 
NS  or  EW  data  set,  the  anomaly  was  declared  a  Category  3,  (2)  if  the  inversion  failed,  the 
anomaly  was  declared  Category  3,  and  (3)  if  an  anomaly  was  present  in  the  NS  transects  only 
(viz.,  no  appreciable  signal  in  the  EW  transects),  it  was  declared  Category  1. 


0  50  100  150  200 

#  DIGS 


Figure  4-36  EM61  MK2  Array  training  data  and  selected  thresholds;  UX- Analyze  analysis.  The 
anomalies  shown  in  green  were  classified  Category  1  based  on  a  comparison  between  data 
collected  along  NS  and  EW  transects. 
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43.2.2.4. 


Performance  Scores  from  IDA 


Scoring  performances  for  the  size-based  analysis  approach  are  reported  in  Table  4-11  and  shown 
graphically  in  Figure  4-37.  A  ROC  chart  is  shown  in  Figure  4-38.  From  a  declared-category 
perspective,  the  category  3  targets  are  plotted  first,  followed  by  categories  5  (red),  4  (orange),  2 
(yellow),  and  finally  1  (green). 

Using  the  established  thresholds,  there  were  no  classification  failures.  All  UXO  were 
categorized  medium  or  high  confidence  UXO  (Category  4  or  Category  5). 


Table  4-11  Performance  Summary:  EM61  MK2  size -based  Analysis 

EM61  Array  UXA _ 


Cultural 

Munition 

Debris 

No 

Contact 

Rock 

Soil 

UXO 

1 

75 

155 

2 

22 

101 

0 

2 

8 

59 

1 

3 

11 

0 

3 

17 

16 

0 

18 

44 

0 

4 

4 

28 

1 

1 

7 

6 

5 

2 

34 

1 

1 

4 

113 

TOTAL  106  292  5  45  167  119 
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Figure  4-37  EM61  MK2  Array  performance  as  a  function  of  classification  category. 
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<SAIC  EM61  MK2  ARRAY  SUBMITTAL  UXA  SC070925.CSV>,  43  934  534  497,  em61,  Test,  All 


Figure  4-38  EM61  MK2  Array  ROC  chart;  analyzed  based  on  size  metric. 

4.3.2.2.5.  Characterization  Plots 

The  RMS  error  between  the  fitted  and  measured  XY  locations  for  seeded  4.2-inch  mortars,  based 
on  UX- Analyze  EM61  MK2  array  inversion  results,  is  0.18m  with  a  standard  deviation  of  0.12m. 
Figure  4-39  plots  the  distance  between  inverted  and  measured  locations  for  UXO  and  clutter, 
segmented  by  discrimination  ranking. 

The  RMS  depth  error  for  seeded  4.2-inch  mortars  is  0.20m  with  a  standard  deviation  of  0.17m. 
Figure  4-40  plots  the  fitted  depth  versus  measured  depth  for  UXO  and  clutter,  segmented  by 
discrimination  ranking. 

Figure  4-41  plots  the  net  polarizability  for  the  first  time  gate  (0.217  ms)  bottom  coil,  for  each  of 
the  119  seeded  4.2in  mortars.  For  the  EM61  MK2  array  data  and  UX- Analyze  solver,  the  mean 
net  polarizability  was  0.751  with  a  standard  deviation  of  0.21 1.  The  corresponding  COV  is 
0.282. 
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Figure  4-39  Differences  between  fitted  and  measured  XY  locations;  EM61  MK2  array  with  size 
based  analysis 
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Figure  4-40  Fitted  versus  measured  depth  of  burial;  EM61  MK2  array  with  size  based  analysis 
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Figure  4-41  Raster  of  the  Zp,  which  is  a  measure  of  size,  for  4.2in  mortars,  EM61  MK2  array 
with  size  based  analysis 


4.3.2.2.6.  Failure  Analyses 

Using  the  thresholds  adopted  for  this  analysis,  there  were  no  false  negatives. 

4.3.2.2.7.  Comments 

Analyzing  the  EM61  MK2  array  data  with  a  size-based  (Zp)  metric  resulted  in  zero  false 
negatives  while  eliminating  49%  of  the  potential  targets. 

Geolocation  errors  and  noise  contributed  to  overlap  in  size  estimates. 

Although  the  size-based  metrics  generated  by  UX-Analyze  were  able  to  separate  the  4.2-inch 
mortars  from  clutter  at  this  site,  the  IDL-based  solver  produced  tighter  polarizations.  Figure  4-42 
compares  primary  versus  secondary  polarizations  for  UXO,  halfshells,  and  baseplates.  The  data 
are  grouped  based  on  labels  received  post  demonstration.  In  the  figure,  symbols  and  color  show 
the  declared  categories.  Diamonds  identify  UXO,  X  are  halfshells,  triangles  are  munitions 
debris,  plus  symbols  are  cultural  and  asterisk  are  used  to  identify  rock/soil  classifications. 
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Figure  4-42  Type  discrimination  as  a  function  of  inversion  approach. 
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4.3.3  GEM-3  ARRAY 


4.3.3. 1.  Data  Presentation,  Description,  and  Pre-processing 

Data  from  the  GEM-3  array  (Figure  4-43)  is  shown  in  Figure  4-44  thru  Figure  4-46.  The  circles 
identify  anomalies  selected  for  analysis  by  the  ESTCP  Program  Office. 

Summary  of  preprocessing  steps  for  GEM-3  Array  data: 

1)  Sensor  cross-calibration  using  ferrite  rods; 

2)  De-median  filter  applied  to  data  to  remove  long  wavelength  drift; 

3)  Inverted  data  from  three  mid-rage  frequencies,  quadrature  only  (in-phase,  three  lowest 
quadrature,  and  three  highest  quadrature  components  discarded  due  to  high  levels  of 
noise)  (Figure  4-47); 

4)  Used  tools  developed  in  Oasis  montaj  to  select  polygons  (spatial  footprint)  about 
anomalies; 

5)  Automated  routines  in  IDF  to  carve  out  data  for  each  anomaly. 

After  preprocessing,  we  ran  inversions  in  batch  and  generated  custom  diagnostic  plots  for  quality 
assurances  purposes. 
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Figure  4-44  GEM  Array  data;  Southwest  area. 
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Figure  4-45  GEM  Array  data;  Southeast  area  1. 
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Figure  4-46  GEM  Array  data;  Southeast  area  2. 
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Figure  4-47  GEM-3  array,  data  down  selection  based  on  noise  characteristics. 


4.3.3.2.  Inversion  Approach 

The  inversion  of  GEM  array  data  is  based  on  a  standard  dipole  model.  The  algorithm  finds  six 
(6)  fitted  parameters:  x,  y,  z,  phi,  theta,  psi,  where  x,  y,  z,  are  the  target  coordinates  (m)  and  phi, 
theta,  psi  are  the  Euler  angles  (degrees).  The  associated  best-fit  betas,  three  polarizations  for 
each  frequency,  are  also  recovered  and  values  for  the  chi-squared  error  and  coherence  are 
calculated.  Follow-on  fits  are  made  with  betas  constrained  to  match  given  values  from  a  library. 
Associated  chi-squared  and  coherence  values  are  found  for  each  library  item,  and  ratios  of 
(constrained  coherence)  /  (un-constrained  coherence)  are  calculated. 

To  avoid  being  trapped  in  local  minima,  unconstrained  inversion  operates  in  two  stages.  The 
first  stage  steps  through  fixed  z  values  from  ground  surface  down  to  1.75m  depth  in  5cm 
intervals  (Figure  4-48).  At  each  z  step,  the  best-fit  target  xy  position  is  found  using  the  “matrix” 
method  in  which  elements  of  the  response  tensor  are  found  through  linear  regression.  This 
permits  fast  run  times  and  experience  has  shown  it  is  robust  against  local  minima.  No 
restrictions  are  placed  on  the  tensor  values:  they  may  be  positive  or  negative,  even  for  quadrature 
data  which  are  normally  only  positive.  Each  channel  is  treated  independently,  so  each  will  have 
a  different  apparent  target  orientation. 
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Red  =  solutions  passed  to  2nd  stage.  Green  =  final  solution  of  2nd  stage. 


Figure  4-48  Example  graph  of  the  error  as  a  function  of  depth  during  the  initial  solver  stage; 
GEM  array  data. 

The  10  best  solutions,  judged  by  chi  square  value,  are  used  as  starting  points  in  the  2nd  stage. 
Before  submitting  these  data,  euler  angles  must  be  found.  It  was  discovered  that  using  arbitrary 
angles  can  lead  to  failure  of  the  solver  due  to  local  minima,  so  a  special  optimization  step  is 
performed  separately  on  each  of  the  10  solutions.  In  this  step,  “best-fit”  Euler  angles  are  found 
through  minimization  of  off-diagonal  elements  in  the  response  tensor,  across  all  channels. 
Starting  points  for  this  optimization  are  the  apparent  Euler  angles  from  each  channel’s  tensor. 

The  second  stage  searches  all  6  fitted  parameter  values:  x,  y,  z,  phi,  theta,  psi,  through  downhill 
simplex  minimization.  After  cycling  through  all  10  of  the  start  points  from  the  1st  stage,  the  best- 
ever  solution  is  saved. 

It  was  found  that  making  an  adjustment  to  In-phase  data  improved  performance,  based  on 
available  ground  truth.  The  adjustment  consists  in  removing  the  local  mean  from  inphase  data  at 
each  measurement  point:  At  each  point,  mean  inphase  across  all  channels  is  found  and 
subtracted  from  inphase  data  at  that  point.  Parameter  search  is  then  performed,  and  finally,  best- 
fit  betas  are  determined  at  the  end  through  regression  on  the  full  unadjusted  data  set. 

Summary  of  solver  steps  used  for  GEM-3  array  data: 

•  Standard  Dipole  Magnetic  Polarization  Model 

•  First  stage: 

-  Step  through  Z  (distance  below  array)  in  5cm  increments. 

-  Time  lag  added  as  fit  parameter  to  compensate  for  dynamic  response. 

-  Search  target  XY  nonlinearly,  find  tensor  elements  linearly. 

-  Find  Euler  angles  to  minimize  off-diagonals  across  all  gates. 

-  Save  10  best  fits  judged  by  chi-square  (different  Z  values  &  lags). 

These  are  start  points  for  following  stages. 

•  Second  stage: 

-  Search  x,  y,  z,  phi,  theta,  psi  all  non-linearly,  find  axial  betas  linearly. 

-  Axial-symmetry  not  assumed. 

•  Third  stage: 

-  Library  of  fixed  beta  values  from  21  UXO  in  GPO  with  high  SNR. 
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-  Search  x,  y,  z,  phi,  theta  non-linearly  (axial  symmetry  imposed). 

-  Fresh  fit  with  each  library  item  using  10  restarts. 

-  Ratio  of  best  library  fit  coherence  /  unconstrained  fit  coherence  used  in 

discrimination.  This  ratio  is  always  <  1 . 


4.3.3.3.  Labeled  data  -  threshold  selection 

Classification  of  the  GEM-3  array  data  was  based  on  results  shown  in  Figure  4-49.  Here,  we 
plot  coherence  ratio  as  a  function  of  fit  mismatch.  We  chose  the  coherence  ratios  instead  of  the 
X2  error  ratios  because  is  tends  to  produce  better  separation  between  TOI  and  non-TOI  classes. 
The  solid  black  line  limits  to  the  mean  value  of  the  coherence  ratio  for  high  SNR  4.2-inch 
mortars.  The  dashed  line  is  shifted  down  from  the  mean  by  10  standard  deviations,  normalized 
by  variance.  The  decision  metric  is  simply  the  distance  from  mean  TOI  response  (solid  black 
line),  in  units  of  coherence  ratio  (function  of  mismatch).  Using  these  labeled  data,  we  set  the 
following  thresholds  (Figure  4-50): 

Category  1:  <  -10 

Category  2:  -10  to  -3 

Category  3:  fit  error  greater  than  50% 

Category  4:  -3  to  0 
Category  5:  >0 

Mitigating  factors  included  the  following:  (1)  if  the  fit  depth  was  greater  than  lm,  then  Category 
2  or  4  based  on  coherence  ratio,  and  (2)  if  the  inversion  failed,  the  anomaly  was  declared 
Category  3. 
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coherence  ratio  (libBest/unconstrained) 


#  UXO  training  set 
o  UXO  GRO  in  library 
x  UXO  GRO  not  library 
a  UXO  Open  field 
^  Cultural 
O  Munitions  Debris 
Half  Round 
Corner  Stake 
Soils 

No  Contact 
o  Rock 


Fit  mismatch:  sqrt(l-corr(data;model)A2) 


Figure  4-49  GEM-3  Array  training  data.  Top  -  Response  of  4.2-inch  mortars  from  GPO. 
Bottom  -  Discrimination  metrics  for  all  training  data. 
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Figure  4-50  GEM-3  Array  training  data  and  selected  thresholds. 


4.3.3.4.  Performance  Scores  from  IDA 

Scoring  performances  for  the  GEM-3  array  analysis  are  reported  in  Table  4-12  and  shown 
graphically  in  Figure  4-51.  A  ROC  chart  is  shown  in  Figure  4-52. 

Using  the  established  thresholds,  there  was  one  False  Negative.  Anomaly  #135  was  declared 
high  confidence  clutter  when  in  fact  it  was  an  emplaced  4.2-inch  mortar.  Anomaly  #133  was 
declared  Category  2  (low  confidence  clutter)  but  was  also  a  seeded  UXO. 
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Table  4-12  Performance  Summary:  GEM3  Array  Constrained/Unconstrained  Analysis 


GEM -3  Array 


Cultural 

Munition 

Debris 

No 

Contact 

Rock 

Soil 

UXO 

1 

104 

317 

5 

17 

21 

1 

2 

4 

44 

0 

5 

15 

1 

3 

18 

37 

11 

23 

97 

1 

4 

1 

20 

0 

5 

7 

38 

5 

0 

5 

0 

0 

4 

78 

TOTAL  127  423  16  50  144  119 


GEM  Array  Scoring  Performance 


□  Cultural 

□  Munition  Debris 

□  No  Contact 

□  Rock 

□  Soil 
■  UXO 


Figure  4-5 1  GEM-3  Array  performance  as  a  function  of  classification  category. 
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<SAIC  GEMTADS  SUBMITTAL  IDL  SC070925.CSV>,  528  5  1061  33,  gem,  Test.  All 


Figure  4-52  GEM-3  Array  ROC  chart;  analyzed  comparing  library-constrained  and 
unconstrained  inversions. 

4.3.3.5.  Characterization  Plots 

The  RMS  error  between  the  fitted  and  measured  XY  locations  for  seeded  4.2-inch  mortars  is 
0.17m  with  a  standard  deviation  of  0.10m.  Figure  4-53  plots  the  distance  between  inverted  and 
measured  locations  for  UXO  and  clutter,  segmented  by  discrimination  ranking. 

The  RMS  depth  error  for  seeded  4.2-inch  mortars  is  0.15m  with  a  standard  deviation  of  0.16m. 
Figure  4-54  plots  the  fitted  depth  versus  measured  depth  for  UXO  and  clutter,  segmented  by 
discrimination  ranking. 

Figure  4-55  plots  the  net  polarizability  for  quadrature  response  at  570  Hz,  for  each  of  the  1 19 
seeded  4.2in  mortars.  For  the  GEM-3  array  data,  the  mean  net  polarizability  was  2.555  with  a 
standard  deviation  of  0.218.  The  corresponding  COV  is  0.097. 
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Figure  4-53  Differences  between  fitted  and  measured  XY  locations;  GEM-3  array 
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Measured  Depth  (m)  Measured  Depth  (m)  Measured  Depth  (m) 
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0.20  0.23 

0.17  0.26 


Figure  4-54  Fitted  versus  measured  depth  of  burial;  GEM-3  array 
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Figure  4-55  Raster  of  the  Xp,  which  is  a  measure  of  size,  for  4.2in  mortars,  GEM-3  array. 

Dashed  lines  are  ±  one  standard  deviation. 

4.3.3.6.  Failure  Analyses 

Anomaly  #135  was  a  seeded  4.2-inch  mortar  but  was  classified  high  confidence  clutter  (Category 
1)  using  the  constrained/unconstrained  approach.  A  reexamination  of  the  field  data  revealed  that 
one  pass  of  the  array  had  elevation  offsets  relative  to  nearby  transects  (Figure  4-56).  The  corrupt 
elevation  (z)  readings  gradually  returned  to  normal  after  traveling  100m.  The  GEM-3  data  were 
interleaved  to  increase  spatial  sampling.  The  poor  z  positioning  of  these  transects  of  data  altered 
the  inverted  polarizations  such  that  it  was  not  fit  well  using  our  library  representing  4.2-inch 
mortars.  Anomaly  #133  was  affected  by  the  poor  elevation  data  as  well  and  was  categorized  low 
confidence  clutter  (Category  2). 
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Measured  Z 


Measured  Z 


150m 


Color  scale  ranges  from 
125.3m  to  125.5m 


Figure  4-56  Contoured  elevation  data  for  GEM-3  sensor  in  vicinity  of  the  single  false  negative 
(#135). 
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4.3.4  MAGNETIC  ARRAY  DATA 


4.3.4.I.  Data  Presentation,  Description,  and  Pre-processing 

Data  from  the  magnetic  array  (Figure  4-57)  is  shown  in  Figure  4-58  thru  Figure  4-60.  The 
circles  identify  anomalies  selected  for  analysis  by  the  ESTCP  Program  Office. 


Figure  4-57  Magnetic  array 
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Figure  4-58  Magnetic  array  data;  Southwest  area. 
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Figure  4-59  Magnetic  array  data;  Southeast  area  1. 
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Figure  4-60  Magnetic  array  data;  Southeast  area  2. 
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4.3A.2.  Inversion  Approach 

The  magnetic  dipole  model  for  this  method  inverts  for  the  location  (Xo,  Yo,  Zo),  magnetic  moment 
(radius  of  an  equivalent  sphere),  and  orientation  angles  ( declination  and  inclination).  The  model 
used  here  first  fits  the  shape  of  the  footprint,  then  its  magnitude.  Initial  guesses  for  the  fit 
parameters  are  determined  internally  within  the  code  based  on  the  measured  signature.  The 
goodness  of  fit  metric,  e ,  is  the  squared  correlation  of  data  and  model.  By  definition,  the  dipole 

fit  error  metric  is  equal  to  Vl  -  £  ■ 

43.4.3.  Labeled  data  -  threshold  selection 

Classification  of  the  magnetic  array  data  was  based  on  the  apparent  size  (scaled  magnetic 
moment).  Figure  4-61  presents  fitted  results  for  all  labeled  data.  In  the  figure,  the  data  are 
prioritized  according  to  apparent  size  and  color  coded  according  to  the  actual  labels.  Based  upon 
these  labeled  data,  we  established  the  following  thresholds: 

Category  1:  <0.055 
Category  2:  0.055  to  0.069 
Category  3:  fit  error  greater  than  60% 

Category  4:  0.069  to  0.10 
Category  5:  >0.10 

The  threshold  between  categories  2  and  4  was  set  based  on  first  UXO  encountered.  The 
thresholds  between  categories  1  and  2  and  between  4  and  5  were  set  to  accommodate  observed 
uncertainty. 

Mitigating  factors  included  the  following:  (1)  if  the  inversion  failed,  the  anomaly  was  declared 
Category  3. 

43.4.4.  Performance  Scores  from  IDA 

Scoring  performances  for  the  magnetic  analysis  are  reported  in  Table  4-13  and  shown 
graphically  in  Figure  4-62.  A  ROC  chart  is  shown  in  Figure  4-63. 
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Figure  4-61  Magnetic  array  training  data  and  selected  thresholds;  UX- Analyze  analysis. 


Table  4-13  Performance  Summary  Magnetic  Array 


MAG  Array 


Cultural 

Munition 

Debris 

No 

Contact 

Rock 

Soil 

UXO 

1 

60 

258 

2 

18 

22 

0 

2 

4 

41 

1 

3 

12 

3 

3 

35 

63 

2 

42 

56 

0 

4 

1 

33 

0 

3 

15 

45 

5 

1 

23 

1 

2 

8 

70 

TOTAL  101  418  6  68  113  118 
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MAG  MK2  Array  Scoring  Performance 


□  Cultural 

□  Munition  Debris 

□  No  Contact 

□  Rock 

□  Soil 
■  UXO 


Anomaly  Count 

Figure  4-62  Magnetic  Array  performance  as  a  function  of  classification  category. 
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Figure  4-63  Magnetic  Array  ROC  chart 

4.3.4.5.  Characterization  Plots 

The  RMS  error  between  the  fitted  and  measured  XY  locations  for  seeded  4.2-inch  mortars,  based 
on  magnetic  array  inversion  results,  is  0.14m  with  a  standard  deviation  of  0.10m.  Figure  4-64 
plots  the  distance  between  inverted  and  measured  locations  for  UXO  and  clutter,  segmented  by 
discrimination  ranking. 

The  RMS  depth  error  for  seeded  4.2-inch  mortars  is  0.098m  with  a  standard  deviation  of  0. 12m. 
Figure  4-65  plots  the  fitted  depth  versus  measured  depth  for  UXO  and  clutter,  segmented  by 
discrimination  ranking. 

Figure  4-66  plots  the  fitted  size,  i.e.,  scaled  magnetic  moment,  for  each  of  the  119  seeded  4.2in 
mortars.  For  the  magnetic  array  data,  the  mean  fitted  size  was  0. 105  with  a  standard  deviation  of 
0.025.  The  corresponding  COV  is  0.24. 
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Figure  4-64  Differences  between  fitted  and  measured  XY  locations;  magnetic  array 
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Figure  4-65  Fitted  versus  measured  depth  of  burial;  magnetic  array 
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Figure  4-66  Raster  of  the  fitted  size  for  4.2in  mortars,  magnetic  array 

43.4.6.  Failure  Analyses 


Using  the  thresholds  adopted  for  this  analysis,  there  were  no  false  negatives. 


Anomalies  149,  170,  and  197,  all  seeded  4.2-inch  mortars  were  declared  Category  2  based  on 
inverted  apparent  sizes  of  0.066,  0.068,  and  0.061  (Figure  4-67;  Figure  4-68).  We  refit  each  of 
these  anomalies  using  different  solvers  and  polygons  during  post  mortem  analysis  and  recovered 
similar  model  parameters. 
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Figure  4-67  Color  contour  map  of  magnetic  data.  Arrows  identify  three  seeded  targets  that  possessed  fitted  sizes  of  less  than  0.068 
(Category  2  declarations).  Color  scale  ±25nT. 
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Figure  4-68  Anomaly  plots  showing  inverted  features  and  forward  model  for  three  anomalies  4.2-inch  mortars  that  possessed  apparent 
sized  of  less  than  0.07  (Category  2). 
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4.3.5  COMBINED  EM61  MK2  AND  MAGNETIC  ARRAY  FEATURES 


4.3.5.1.  Overview 

For  the  Combined  Magnetic  and  EM  dig  list,  we  inverted  the  EM61  MK2  and  magnetic  array 
data  independently,  as  described  in  Sections  4.3.1  and  4.3.4,  and  combined  the  features  using 
conservative  Boolean  logic  operators.  In  order  to  produce  a  combined  ranking  for  targets 
surveyed  by  either  one  or  both  of  the  two  systems,  decision  metrics  from  (i)  EM  only,  (ii) 
magnetic  only,  and  (iii)  GLRT  classifier  output  using  magnetic  size  and  EM  coherence  ratio  as 
inputs,  were  involved  in  the  final  categorization. 

It  should  be  remembered  that  our  magnetic-only  categories  were  based  solely  on  the  inverted 
size.  The  EM  only  categories  were  based  on  the  coherence  ratio  of  the  library-constrained  fit 
versus  the  unconstrained  fit.  The  EM  decision  metrics  therefore  are  influenced  by  size  and  shape 
estimates.  Based  upon  the  labeled  data,  the  EM  classifications  performed  better  than  the 
magnetic.  Because  of  this,  slightly  more  confidence  was  given  to  the  EM-only  results  in  the 
event  of  different  classifications.  The  combined  category  was  based  on  the  following  Boolean 
rules: 

1)  if  the  anomaly  was  surveyed  by  only  one  instrument,  adopt  the  corresponding  category 

2)  if  the  anomaly  was  surveyed  by  both,  then: 

a.  if  the  EM  only  is  high  confidence  UXO,  then  adopt  the  EM  Category 

b.  if  EM-only  and  magnetic-only  were  each  Category  3  or  4,  then  assign  Category  4 
(reduce  the  number  of  can’t  analyze  declarations) 

c.  otherwise,  adopt  the  combined  GLRT-assigned  category. 

4.3.5.2.  Performance  Scores  from  IDA 

Scoring  performances  for  the  constrained  versus  unconstrained  analysis  approach  are  reported  in 
Table  4-14  and  shown  graphically  in  Figure  4-69.  A  ROC  chart  is  shown  in  Figure  4-70. 

Table  4-14  Performance  Summary  Joint  Magnetic  &  EM61  Array 


JOINT  MAG  &  EM61  ARRAYS 
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EM61  MK2  ARRAY  PLUS  MAG  Scoring  Performance 
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□  No  Contact 
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□  Soil 
■  UXO 


Anomaly  Count 

Figure  4-69  Magnetic  Array  performance  as  a  function  of  classification  category 
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Figure  4-70  Magnetic  Array  ROC  chart 

4.3.5.3.  Failure  Analyses 

Using  the  thresholds  and  logic  adopted  for  this  analysis,  there  were  no  false  negative. 

Anomaly  #192  was  a  seeded  mortar  and  was  ranked  Category  2.  It  was  the  last  Category  2  target 
before  Category  4.  For  this  anomaly,  the  magnetic-only  Category  was  2,  the  EMI-only  Category 
was  2,  and  the  GLRT  combined  Category  for  this  anomaly  was  2.  Anomaly  #420,  which  was  a 
false  negative  failure  of  the  EM61  MK2  array  only  processing  (§4.3.2. 1)  due  to  positioning 
errors,  was  not  a  failure  in  this  analysis  because  the  magnetic-only  analysis  classified  it  as  a 
Category  4  based  on  apparent  size. 
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4.4  In-air  Clutter  Measurements 

After  the  excavations  were  complete,  803  clutter  items  were  re-numbered  (to  protect  their 
identity)  and  shipped  to  Blossom  Point  by  the  Program  Office  (Table  4-15).  Basically,  the  idea 
was  to  collect  data  in  a  controlled  setting  in  order  to  collect  extremely  high-quality  data  that 
could  be  used  to  establish  discrimination  bounds  for  this  collection  of  UXO  and  clutter.  By 
recording  signatures  for  each  object  individually,  at  optimal  coil- target  separations,  the  in-air 
controlled  tests  inherently  removed  all  positioning  errors,  eliminated  any  possible  overlapping 
signatures,  eradicated  motion  noise,  and  did  away  with  any  soil  response. 

Time-domain  EMI  data  were  collected  using  the  G&G  Science  array  developed  under  funding 
from  MM-0601  (Figure  4-71).  It  consists  of  25  transmit  (Tx)/receive  (Rx)  pairs  arranged  in  a 
5x5  fixed-separation  grid.  For  these  tests,  we  used  the  central  9Tx  coils  and  recorded  the  decay, 
using  all  25Rx  coils,  from  0.04msec  to  25msec.  We  performed  dipole  inversions  for  all 
signatures  to  obtain  principal  axis  polarizabilities  (approximately  5%  were  too  small  to  produce 
decent  inversion  results).  The  data  are  of  excellent  quality.  Fits  for  an  example  mortar  and 
large,  medium,  and  small  clutter  are  shown  in  Figure  4-72  through  Figure  4-75  respectively. 

The  net  polarizability,  defined  here  as  Spi  for  t  =  0.04-0.06  msec  (Figure  4-76),  indicates  that 
perfect  discrimination  is  possible  for  this  distribution  of  UXO  and  clutter  using  only  size 
estimates.  As  shown  in  the  figure,  the  4.2in  mortar  target  strength  is  clearly  larger  than  any 
clutter  item. 

Table  4-15  In-air  TEM  measurements  of  Sibert  clutter 
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Figure  4-71  The  prototype  time  domain  EMI  (TEM)  sensor  developed  under  MM-0601  was 
utilized  to  measure  responses  of  all  Sibert  clutter. 
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Figure  4-72  Principal  axis  p  decay  curves  for  4.2in  mortar  (seed  target  42-073).  Six  different 
location/orientation  combinations. 
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Figure  4-73  Principal  axis  P  decay  curves  for  large  clutter  (half  shells) 
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Figure  4-74  Principal  axis  p  decay  curves  for  medium  clutter  (base  plates,  etc.) 
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Figure  4-75  Principal  axis  p  decay  curves  for  small  clutter 


Net  Polarizability  (0.04-0.06  msec) 

Figure  4-76  In-air  target  strength  (net  polarizability,  Dpi  for  t  =  0.04-0.06  msec)  suggests  perfect 
discrimination  is  possible  for  this  collection  of  TOIs  and  clutter.  The  4.2-inch  mortar  target 
strength  is  clearly  larger  than  any  clutter  item 
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5.  Discussion 


We  were  provided  magnetic  and/or  EMI  data  and  target  selections  (ID,  XY),  and  were  asked  to 
discriminate  between  native  clutter  objects  and  4.2in  mortars,  whether  seeded  or  native  (no 
native  4.2in  mortars  recovered).  Because  4.2in  mortars  generally  produce  high  SNR  responses, 
the  primary  problem  became  one  of  discriminating  the  mortars  that  were  deeply  buried  and 
therefore  possessed  broad,  low-amplitude  responses  that  could  be  masked  by  geology.  Data 
shown  in  Figure  5-1  illustrates  the  issue.  The  circles  in  the  figure  identify  the  seeded  4.2in 
mortars.  As  shown  here,  not  all  of  the  high  amplitude  anomalies  are  caused  by  4.2in  mortars 
and,  conversely,  not  all  of  the  small  amplitude  anomalies  are  clutter.  Our  objective  was  to 
identify  as  many  clutter  objects  as  possible  without  misclassifying  any  4.2in  mortars. 

Figure  5-2  shows  photographs  of  items  recovered  during  the  demonstration.  The  largest  clutter 
items  were  the  half  shells,  which  were  presumably  created  when  the  4.2in  shell  casings  split 
open  at  impact. 

Anomalies  were  selected  by  the  Program  Office  based  on  thresholds.  The  thresholds  were  set  to 
be  one-half  of  the  observed  sensor-dependent  magnitude  for  4.2in  mortars  buried  at  their 
maximum  (1  lx)  depth.  Because  of  this  selection  method,  the  number  of  anomalies  identified  for 
analysis  varied  across  data  types  (Figure  5-3). 

The  EM61  cart  data  possessed  variable  time  lag  that  degraded  the  fitted  results  (Figure  5-4).  To 
alleviate  this  problem,  we  added  a  time  lag  parameter  to  the  solver  used  for  the  EM61  cart  data. 

We  processed  and  classified  each  data  type  independently  and  did  not  compare  results.  Because 
of  the  size  difference  in  the  TOI  versus  clutter  at  this  site,  however,  a  significant  fraction  of  the 
targets  were  identified  as  high  confidence  clutter  regardless  of  data  type  or  classification 
approach  (see  Category  1;  Figure  5-5). 

As  discussed  in  §4,  we  based  our  classifications  on  (i)  the  estimated  size  or  (ii)  fit  ratios  using  a 
library  that  consisted  only  of  4.2in  responses.  The  second  approach  inherently  includes  size  and 
shape  information. 

Performance  results  for  size  based  classifications  of  magnetic  and  EMI  data  are  shown  in  Figure 
5-6.  Although  both  approaches  performed  well  (Table  4-4  and  Table  4-6),  the  EMI-based  results 
had  fewer  Can’t  Analyze  declarations  and  better  separation  between  classes  than  the  magnetic 
(as  visually  observed  by  the  slope  of  the  curve  as  well  as  AUC  of  0.97  for  EMI  versus  0.91  for 
magnetic). 

We  analyzed  the  EM61  array  twice.  Different  inversion  routines  and  classification  schemes 
were  used.  As  detailed  above,  one  approach  performed  a  non-linear  simultaneous  inversion  for 
all  model  parameters  and  based  the  classification  on  apparent  size.  The  second  performed  a 
multi-step  inversion  that  involved  stepping  through  z,  non-linear  and  linear  searches,  and 
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schemes  to  avoid  local  minima.  The  classification  metric  for  the  second  approach  involved 
comparisons  of  fit  quality  of  unconstrained  versus  library  constrained  inversions.  The 
approaches  were  indeed  quite  different.  The  performances,  however,  are  strikingly  similar 
(Figure  5-7).  No  significant  differences  in  the  ROC  curves  are  apparent  with  the  notable 
exception  that  the  library-based  approach  had  one  false  negative  (failure).  Because  the  library- 
based  approach  incorporates  size  and  shape  information  (in  the  form  of  the  library),  it  is  more 
sensitive  to  glitches  in  the  data.  Our  post-mortem  analysis  revealed  that  the  false  negative  was 
indeed  caused  by  a  GPS  data  problem  that  went  unnoticed  during  analysis. 

A  performance  comparison  of  cart  versus  array  deployment  methods  for  the  EM61  MK2  sensor 
shows  minor  differences  in  the  shape  of  their  respective  ROC  curves  (Figure  5-8).  There  were, 
however,  differences  of  significance.  First,  the  cart  data  classified  9  TOIs  as  Can’t  Analyze 
(Category  3)  due  to  large  fit  errors.  Second,  the  cart-based  analysis  had  one  false  negative 
failure. 

A  conceptual  model  useful  for  framing  the  problem  is  to  divide  the  ROC  curve  shown  in  Figure 
5-9  into  three  areas;  (A)  can’t  analyze,  (B)  analyzed-and-dig,  and  (C)  analyzed-but-do-not-dig. 
Clearly,  our  objective  is  to  (i)  reduce  the  number  of  ‘can’t  analyze’  declarations,  (ii)  reduce  the 
number  of  analysis  failures  in  the  ‘analyzed-and-dig’  category,  and  (ii)  quantify  the  residual  risk 
in  the  ‘analyzed-but-do-not-dig’  space. 

With  regard  to  the  ‘analyzed-and-dig’  category,  our  post  mortem  analysis  revealed  that  some  of 
the  failures,  and  many  of  the  near  failures,  were  caused  by  spatial  registration  problems.  We 
discovered  problems  with  the  measured  GPS  elevation  and  timing  lags  (Figure  5-10). 

Excluding  the  EM61  cart  data,  the  sources  of  the  ‘Can’t  Analyze’  declarations  turned  out  to  be 
small  fragments,  scrap  metal,  no  contact,  or  soil.  These  anomalies  were  characterized  by  low 
amplitude  responses  and  small  spatial  footprints  and  were  not,  therefore,  well  characterized  by 
our  models  (Figure  5-11,  Figure  5-12). 

As  shown  in  §4.4  In-air  Measurements,  the  4.2in  mortars  had  target  strengths  that  were  clearly 
larger  than  any  of  the  clutter.  In  other  words,  based  on  size  estimates  alone,  these  in-air 
measurements  suggest  perfect  discrimination  is  possible  for  this  distribution  of  UXO  and  clutter. 
Figure  5-13  compares  the  net  polarizability  of  the  in-air  measured  responses  to  the  EM61  array 
field  data.  For  field  data,  the  p  spread  results  in  overlap  between  UXO  and  clutter.  Geolocation 
errors  are  a  likely  factor  in  the  observed  30%  RMS  P  spread.  In  Figure  5-14,  we  approximated 
the  p  error  using  the  mortar  distribution  and  used  it  to  smear  out  the  in-air  distributions.  After 
accounting  for  p  spread,  the  in-air  and  EM61  array  field  data  have  similar  distributions  for  net 
polarizabilities  >0.1.  Differences  for  weak  targets  (Zp  <  0. 1)  are  consistent  with  increasingly 
poor  fit  quality  and  inability  to  classify. 
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In  summary,  it  is  possible  to  separate  4.2-inch  mortars  from  native  clutter  found  at  this  site  using 
geophysical  data  acquired  in  survey  mode: 

1.  Size -based  decisions  worked  fairly  well.  Over  43%  of  the  targets  were  rejected  as  ‘high 
confidence  clutter’  without  failure  using  size-based  decisions. 

2.  GPS  positioning  errors  caused  the  classification  failures  for  the  EM61  array  and  GEM-3 
array  data.  The  single  false  negative  in  the  EM61  cart  analysis  was  caused  by  a  poorly 
defined  spatial  footprint. 

3.  In-air  target  strengths  of  4.2-inch  mortars  are  larger  than  any  clutter  item,  indicating  that 
perfect  discrimination  is  possible,  for  this  set  of  UXO  and  clutter,  using  systems  that 
mitigate  or  eliminate  positioning  errors  and  motion  noise. 


97 


EM61  Array  Data  Color  Scale  0-50mV  10m  grid  lines 


Figure  5-1  EM61  MK2  array  data  from  the  geophysical  prove  out  (left)  and  southeast  1  area 
(right) 
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Figure  5-2  Photographs  of  objects  recovered  from  the  Sibert  demonstration  area 
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Figure  5-3  Number  of  anomalies  identified  varied  across  data  sets 


Single  Cart  Towed  Array 


(meters) 

Figure  5-4  EM61  MK2  data  from  the  geophysical  prove  out  area  for  the  cart  (bottom  left)  and 
array  (bottom  right).  As  observed  in  the  fitted  results  (top),  the  array  produced  a  tight  clustering 
of  polarizations  (black  is  UXO  and  red  represents  half  shells)  but  the  cart  did  not.  To  fix  this 
problem,  we  added  a  time  lag  parameter  to  the  cart  solver. 
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□  1  High  Confidence  Non-TOI  (NO  DIG) 

□  2  Low  Confidence  NonTOI  (DIG) 

□  3  Can't  Analyze  (DIG) 

□  4  Low  Confidence  TOI  (DIG) 
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Figure  5-5  Comparison  of  classification  results 
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Figure  5-6  Performance  comparison  of  size  based  classification  for  magnetic  and  EMI  array  data 
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No  significant  improvement  over  size  metric 
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Figure  5-7  Performance  comparison  of  size-  versus  library-based  classification  for  EMI  array 
data 
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No  significant  difference  between  platforms 


Figure  5-8  Performance  comparison  of  cart  versus  array  EMI  data 
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SAIC  Analysis:  EM61  MK2  Array  (library  analysis) 
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Figure  5-9  The  ROC  curves  can  be  divided  into  three  conceptual  domains 
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Figure  5-10  A  post  mortem  failure  analysis  revealed  that  sources  of  analysis  errors  were  caused 
primarily  by  GPS  errors 
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Figure  5-11  Example  EMI  anomalies  classified  Can’t  Analyze  (Category  3) 
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Figure  5-12  Example  magnetic  anomalies  classified  Can’t  Analyze  (Category  3) 
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Mel  Polarizability  (gate  i) 

Figure  5-13  Net  polarizabilities  for  in-air  versus  EM61  array  field  data 
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Figure  5-14  After  accounting  for  b  spread,  the  in-air  and  EM61  array  field  data  have  similar 
distributions  for  net  polarizabilities  >0.1. 
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6.  Cost  Assessment 


6.1  Cost  Reporting 

Cost  categories  for  the  overall  ESTCP  Discrimination  Pilot  Program  include  mobilization,  field 
survey,  data  analysis,  demobilization,  and  reporting.  Cost  categories  for  this  data  analysis 
demonstration  are  data  analysis  and  reporting. 

The  data  analysis  phase  consists  of  a  number  of  distinct  and  sequential  operations.  The 
operations  include  pre-processing,  anomaly  extraction,  characterization  (inversion), 
classification,  and  documentation.  The  hours  and  costs  were  tracked  throughout  the 
demonstration  and  are  shown  in  Table  6-1.  The  reported  costs  assume  an  average  labor  rate  of 
$115  per  hour. 

The  magnetic  array  data  required  the  fewest  number  of  hours  and  lowest  cost  per  target.  The 
EM61  Cart  data  required  more  pre-processing  than  the  EM61  array  data  because  it  was  collected 
in  a  number  of  small  grids  with  overlapping  turns  instead  of  large  sorties  without  overlaps  and 
because  the  organization  of  the  delivered  data  required  more  manipulation  than  did  the  array 
data.  TheEM61  array  data  analyzed  with  our  IDL  solver  required  more  time  during  the 
characterization  phase  than  that  required  by  UX- Analyze  because  the  IDL  routine  does  a  much 
more  thorough  job  of  avoiding  local  minima  and  has  numerous  restarts  and  significantly  more 
logic  loops. 


Table  6-1  Cost  Summary 


Cost  Summary 

Hours 

Cost 

Number  of 
Targets 

Cost  per 
Target 

Pre-Demo  Testing  Prep 

650 

74,750 

EM61  Cart 

108 

12,420 

671 

$ 

18.51 

MAG  Array 

66 

$ 

7,590 

1007 

$ 

7.54 

GEM-3  Array 

94 

10,810 

1077 

$ 

10.04 

EM61  Array  (UX- Analyze) 

84 

$ 

9,660 

908 

$ 

10.64 

EM61  Array  (IDL  Solver) 

126 

14,490 

908 

$ 

15.96 

Demonstration  Report  and  Meetings 

300 

34,500 

Total 

1428 

164,220 

4571 
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6.2  Cost  Analysis 


6.2.1  Cost  Basis 

The  cost  basis  for  this  demonstration  is  the  labor  hours  spent  characterizing,  classifying,  and 
documenting  the  discrimination  results. 

6.2.2  Cost  Drivers 

The  primary  cost  driver  for  this  technology  is  the  amount  of  time  spent  processing  and 
interpreting  each  anomaly.  If  targets  are  isolated,  leveled,  and  of  high  data  quality  then  the  cost 
is  driven  by  inversion  times  and  time  spent  making  the  classification  decision.  If  a  lot  of 
preprocessing  is  required  to  identify  spatial  footprints,  to  remove  turnarounds,  to  recognize 
database  problems,  or  to  re-level  the  data,  then  the  cost  is  driven  by  these  later  factors  and  the 
inversion  time  becomes  much  less  important.  For  the  most  part,  the  data  collected  at  Sibert  had 
anomalies  that  were  isolated. 
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7.  Implementation  Issues 


7.1  Regulatory  and  End-user  Issues 

The  ESTCP  Program  Office  established  an  Advisory  Group  to  facilitate  interactions  with  the 
regulatory  community  and  potential  end-users  of  this  technology.  Members  of  the  Advisory 
Group  include  representatives  of  the  US  EPA,  State  regulators,  Corps  of  Engineers  officials,  and 
representatives  from  the  services.  The  ESTCP  staff  worked  with  the  Advisory  Group  to  define 
goals  for  this  Program  and  developed  Project  Quality  Objectives.  As  the  analyzed  data  from  the 
demonstrations  become  available,  the  Advisory  Group  assisted  in  developing  a  validation  plan. 
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